Conversion path performance measures and reports

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing conversion path performance measures and reports. In one aspect, user interaction data are obtained, were the user interaction data specify user interactions for a plurality of conversions. User interactions that are associated with each conversion are selected from the user interaction data, where the associated user interactions for each conversion are user interactions with a converting user during the conversion cycle for the conversion. Using the user interaction data for the selected user interactions, a quantity of user interactions that are associated with each conversion and occurred during the conversion cycle for the conversion are determined. In turn, conversion path performance measures are computed and reports specifying the conversion path performance measures are generated.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. §119(e) of U.S.Patent Application No. 61/316,657, entitled “Conversion Path PerformanceMeasures and Reports,” filed Mar. 23, 2010, which is incorporated hereinby reference in its entirety.

BACKGROUND

This specification relates to determining content performance measuresand providing performance reports.

The Internet provides access to a wide variety of resources. Forexample, video and/or audio files, as well as web pages for particularsubjects or particular news articles are accessible over the Internet.Access to these resources presents opportunities for advertisements tobe provided with the resources. For example, a web page can includeadvertisement slots in which advertisements can be presented. Theseadvertisements slots can be defined in the web page or defined forpresentation with a web page, for example, in a pop-up window.

Advertisement slots can be allocated to advertisers through an auction.For example, advertisers can provide bids specifying amounts that theadvertisers are respectively willing to pay for presentation of theiradvertisements. In turn, an auction can be performed and theadvertisement slots can be allocated to advertisers according to theirbids. When one advertisement slot is being allocated in the auction, theadvertisement slot can be allocated to the advertiser that provided thehighest bid or a highest auction score (e.g., a score that is computedas a function of a bid and/or an advertisement quality measure). Whenmultiple advertisement slots are allocated in a single auction, theadvertisement slots can be allocated to a set of bidders that providedthe highest bids or have the highest auction scores.

Advertisement management accounts can enable advertisers to specifykeywords and corresponding bids that are used to control allocation oftheir advertisements. The advertiser can also track the performance ofadvertisements that are provided using the keywords and correspondingbids. For example, an advertiser can access the advertisement managementaccount and view performance measures corresponding to the advertiser'sadvertisements that were distributed using each keyword. In turn, theadvertiser can adjust settings that control the allocation ofadvertisements and compare the performance measures for theadvertisements that are allocated using the new settings.

SUMMARY

Content providers (e.g., advertisers) are provided reports that specifyuser interactions with content that is distributed to users for thecontent providers. In some implementations, the reports that areprovided to a particular content provider specify conversion pathperformance measures that measure user interactions with contentprovided by the advertiser and that occurred prior to a conversion. Forexample, the conversion path performance measures can specify a totalquantity of interactions between the advertiser's content and a userprior to the user performing an action that constitutes a conversion.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof obtaining user interaction data for a plurality of conversions, wherethe user interaction data specify, for each of the conversions, userinteractions with content items that are associated with one or moretargeting keywords. The targeting keywords control distribution of thecontent items and the user interactions for each conversion are one ormore presentations of content items during a conversion cycle for theconversion and zero or more user selections of the content items thatoccurred during the conversion cycle. The conversion cycle for eachconversion is a period preceding a user interaction that constitutes aconversion. A first time at which each conversion occurred isdetermined. A reference user interaction that occurred prior toselection of a last selected content item is selected, where the lastselected content item is a content item that was last selected prior tothe conversion. A second time at which the reference user interactionoccurred is determined. A time lag is computed for a content provider,where the time lag is a measure of time between the reference userinteraction and the conversion. Data that cause presentation of the timelag are provided. Other embodiments of this aspect include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices.

These and other embodiments can each optionally include one or more ofthe following features. The content item can be an advertisement, and atime lag for the content provider can be determined for an advertiser asa measure of time between an initial advertisement impression and theconversion, where the initial advertisement impression is an earliestadvertisement impression during the conversion cycle. Other embodimentsof this aspect include corresponding systems, apparatus, and computerprograms, configured to perform the actions of the methods, encoded oncomputer storage devices.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods in which user interaction dataare obtained for a plurality of conversions, where the user interactiondata specify, for each of the conversions, corresponding userinteractions with content items that are associated with one or moretargeting keywords. User interactions are selected from the userinteraction data for each conversion, where the user interaction dataare associated with the conversion. The associated user interactions areuser interactions with a converting user during the conversion cycle.Using the user interaction data for the selected user interactions, aquantity of user interactions that are associated with the conversionand occurred during the conversion cycle for the conversion aredetermined. A path length measure that represents an aggregate quantityof user interactions that occurred during conversion cycles for theconversions is computed and data that cause presentation of the pathlength measure are provided. Other embodiments of this aspect includecorresponding systems, apparatus, and computer programs, configured toperform the actions of the methods, encoded on computer storage devices.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods in which user interaction datafor a plurality of conversions are obtained, where the user interactiondata specify, for each of the conversions, corresponding userinteractions with content items that are associated with one or moretargeting keywords, and where the targeting keywords controldistribution of the content items. The corresponding user interactionsfor each conversion are one or more presentations of content itemsduring a conversion cycle for the conversion and zero or more userselections of the content items that occurred during the conversioncycle, the conversion cycle for each conversion being a period precedinga user interaction that constitutes a conversion and ending with theconversion. A set of user interactions are selected for each conversion,each of the selected user interactions being user interactions thatoccurred during the conversion cycle for the conversion. A conversionpath is generated for each conversion, the conversion path specifying anorder in which the user interactions from the set of user interactionsoccurred. The conversion path further specifies keywords that causedpresentation of content items with which the user interactions occurred.Data that cause presentation of at least one conversion path for atleast one conversion is provided. Other embodiments of this aspectinclude corresponding systems, apparatus, and computer programs,configured to perform the actions of the methods, encoded on computerstorage devices.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods in which user interaction datafor a plurality of conversions are obtained, where the user interactiondata specify, for each of the conversions, corresponding userinteractions with content items that are associated with one or moretargeting keywords. The targeting keywords control distribution of thecontent items and the corresponding user interactions for eachconversion are one or more presentations of content items during aconversion cycle for the conversion and zero or more user selections ofthe content items that occurred during the conversion cycle. Theconversion cycle for each conversion is a period preceding a userinteraction that constitutes a conversion. A quantity of assist userinteractions are determined for each conversion, where the quantity ofassist user interactions includes at least one of a quantity of assistimpressions and a quantity of assist clicks. The quantity of assistimpressions is a quantity of content item presentations that occurredduring a conversion cycle for the conversion, and the quantity of assistclicks is a quantity of user selections of content items that occurredduring a conversion cycle for the conversion and prior to a last clickfor the conversion, where the last click is a last user selection of acontent item preceding the conversion. A measure of assist userinteractions is computed, where the measure of assist user interactionsis an aggregate measure of assist user interactions for the conversions.Other embodiments of this aspect include corresponding systems,apparatus, and computer programs, configured to perform the actions ofthe methods, encoded on computer storage devices.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods in which user interaction datais obtained for a plurality of conversions, where the user interactiondata specify, for each of the conversions, corresponding userinteractions with content items that are associated with one or moretargeting keywords. The targeting keywords control distribution of thecontent items and the corresponding user interactions for eachconversion are one or more presentations of content items during aconversion cycle for the conversion and zero or more user selections ofthe content items that occurred during the conversion cycle. Theconversion cycle for each conversion is a period preceding a userinteraction that constitutes a conversion. Assist keywords that causedpresentation of each of identified content item for each conversion areidentified. Each assist keyword is associated with at least one of anassist user interaction that occurred during a conversion cycle for theconversion and at least one assist keyword being associated with a userinteraction that occurred prior to presentation of a last clickedcontent item for the conversion. An assisted conversion count isdetermined for each assist keyword, where an assisted conversion countspecifies a quantity of conversions with which the assist keyword isassociated. An assisted conversion measure is computed using theassisted conversion counts for the assist keywords. An assistedconversion is a conversion for which at least one content item waspresented to a converting user and zero or more of the presented contentitems were selected by the converting user prior to a last click. Aconverting user is a user that performed the user interaction thatconstitutes a conversion, where a last click is a last selection of acontent item prior to a conversion. Other embodiments of this aspectinclude corresponding systems, apparatus, and computer programs,configured to perform the actions of the methods, encoded on computerstorage devices.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods in which user interaction datafor a plurality of conversions are obtained, where the user interactiondata specify, for each of the conversions, a set of corresponding userinteractions with content items that are associated with one or moretargeting keywords. The targeting keywords control distribution of thecontent items, and the set of corresponding user interactions for eachconversion are one or more presentations of content items during aconversion cycle for the conversion and zero or more user selections ofthe content items that occurred during the conversion cycle. Theconversion cycle for each conversion is a period preceding a userinteraction that constitutes a conversion. A request is received for afirst measure of user interactions for a targeting keyword. The requestspecifies that the targeting keyword is a keyword that causedpresentation of a candidate content item with which a particular userinteraction occurred prior to a last click, where the last click is alast selection of a content item by a converting user prior to aconversion by the converting user. A first set of user interaction datais selected where the first set of user interaction data specifyconversions that are associated with the targeting keyword and furtherspecify user interactions that occurred during conversion cycles for theconversions. Using the first set of user interaction data, the firstmeasure of user interactions is computed for the targeting keyword,where the first measure of user interactions is a measure of userinteractions for one or more conversions for which the targeting keywordcaused presentation of the candidate content item to the convertinguser. A request is received for a second measure of user interaction,where the second request specifies that the second measure is to becomputed with user interaction data for the one or more conversions. Thesecond measure of user interactions is computed for the one or morekeywords using the user interaction data for the one or moreconversions. Other embodiments of this aspect include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods in which user interaction datafor a plurality of conversions are obtained from each of a plurality ofdifferent user interaction data sources. The user interaction data fromat least one of the user interaction data sources specify, for each ofthe conversions, a set of user interactions with content items that areassociated with one or more targeting keywords, wherein the targetingkeywords control distribution of the content items. The user interactiondata from at least one other user interaction data source specify, foreach of the conversions, a set of user interactions with a web page thatis provided by a content provider for which distribution of the contentitems is controlled using the targeting keywords. Each set of userinteractions for each conversion specify one or more user interactionsduring a conversion cycle for the conversion, where the conversion cyclefor each conversion is a period preceding a user interaction thatconstitutes a conversion. A request is received for conversion pathperformance measures, where the request specifies two or more differentsources of user interaction data from which user interaction data is tobe used to compute the conversion path performance measures. Anaggregate set of user interaction data are selected from the userinteraction data from the two or more different sources of userinteraction data. The conversion path performance measures are computedusing the aggregate set of user interaction data. Data that causepresentation of the conversion path performance measures are provided.Other embodiments of this aspect include corresponding systems,apparatus, and computer programs, configured to perform the actions ofthe methods, encoded on computer storage devices.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods in which one or more userinteraction logs are processed to obtain respective user interactiondata for one or more user interactions. A first distributed data tableis updated using the respective user interaction data for the userinteractions. The first distributed data table includes rows indexed andsorted by strings representing unique pairs of advertiser identifier anduser identifier that are associated with stored user interactions in thefirst distributed data table. Each row includes a plurality of orderedcells, where each cell corresponds to a respective user interaction typeand contains the user interaction data for stored user interactions ofsaid user interaction type. The user interaction data for each userinteraction is stored with a respective timestamp associated with theuser interaction. The first distributed data table is queried toidentify conversions that have occurred within a specified time windowbased on the user interaction types and timestamps of the stored userinteractions. In response to the query, the user interaction data isreceived for each identified conversion, as well as respective userinteraction data of all stored user interactions associated with a sameadvertiser identifier and user identifier pair as the conversion andhaving occurred within a specified time period prior to the identifiedconversion. A new row is created for each identified conversion in asecond distributed data table, the new row being indexed with a uniqueconversion identifier and including all of the received user interactiondata for the identified conversion. Other embodiments of this aspectinclude corresponding systems, apparatus, and computer programs,configured to perform the actions of the methods, encoded on computerstorage devices.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Content providers are provided new conversion pathperformance measures that enable the content providers to analyze userinteractions with their content prior to a conversion by the user. Userinteraction data is more efficiently processed by arranging the userinteraction data in a conversion centered distributed data table. Theconversion centered distribution table enables scalability throughconvenient data addition and updates of particular portions of the datatables without requiring a full update of the data table. For example,by organizing user interaction data into the distributed data tableindexed and sorted by conversions, data of all user interaction eventsassociated with a conversion can be stored in a single row in thedistributed data table. This enables generation of conversion relatedmetrics to be performed in a distributed computing environment whereprocessing of individual conversions can be parallelized.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example environment in which anadvertisement management system manages advertising services.

FIG. 2 is a flow chart of an example process for providing a conversionperformance report.

FIG. 3A is a flow chart of an example process for determining time lagsfor advertisement conversions.

FIGS. 3B and 3C are screen shots of example user interfaces that displayinformation for time lags for advertisement conversions.

FIG. 4A is a flow chart of an example process for determining pathlengths for advertisement conversions.

FIG. 4B is a screen shot of an example user interface for providing pathlength measures for conversions.

FIG. 5A is a flow chart of an example process for determining top pathsfor advertisement conversions.

FIG. 5B is a screen shot of an example user interface for providing forproviding top path measures for conversions.

FIG. 6A is a flow chart of an example process for tracking assist userinteractions in association with conversions.

FIG. 6B is a screen shot of an example user interface for providingassist click and assist impression associated measures.

FIG. 7A is a flow chart of an example process for computing assistedconversion measures.

FIG. 7B is a screen shot of an example user interface for providingassisted conversion measures.

FIG. 8A is a flow chart of an example process for drilling intoconversion path performance measures.

FIG. 8B is a screen shot of an example user interface for providingdrilled down conversion path performance measures.

FIG. 9A is a screen shot of an example user interface for providingconversion path performance measures derived from multiple sources ofuser interactions.

FIG. 9B is a flow chart of an example process for computing conversionpath performance measures using user interaction data from multipledifferent sources.

FIG. 10A is a block chart of an example data processing pipeline forproviding conversion performance reports.

FIG. 10B is a flow chart of an example process for organizing individualuser interaction data first into an advertiser-user centered datastorage model and then into a conversion centered data storage model.

FIG. 10C is a flow chart of an example process for adding new userinteraction data in a conversion centered data storage model.

FIG. 10D is a flow chart of an example process for generating aggregateddata tables utilizing the user interaction data that are stored inaccordance with the conversion centered data storage model.

FIG. 10E is a flow chart of an example process for providing conversionperformance reports based on pre-stored aggregated data.

FIG. 11 is a block diagram of an example computer system.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

Content providers (e.g., advertisers) are provided reports that specifyuser interactions with content that is distributed to users for thecontent providers. In some implementations, the reports that areprovided to a particular content provider specify conversion pathperformance measures that measure user interactions with contentprovided by the advertiser and that occurred prior to a conversion. Forexample, the conversion path performance measures can specify a totalquantity of interactions between the advertiser's content and a userprior to the user performing an action that constitutes a conversion.

A conversion occurs when a user performs a specified action, and aconversion path is a set of user interactions by a user prior to aconversion by the user. What constitutes a conversion may vary from caseto case and can be determined in a variety of ways. For example, aconversion may occur when a user clicks on an advertisement, is referredto a web page, and then consummates a purchase before leaving that webpage.

Actions that constitute a conversion can be specified by each advertiseron an advertiser by advertiser basis. For example, each advertiser canselect, as a conversion, one or more measurable/observable user actionssuch as, for example, downloading a white paper, navigating to at leasta given depth of a website, viewing at least a certain quantity of webpages, spending at least a predetermined amount of time on a website orweb page, or registering on a website. Other actions that constitute aconversion can also be used. As used throughout this document, aconversion occurs upon the occurrence of the final action that is usedto define a conversion. For example, if a user visiting 5 web pagesdefines a conversion, the conversion occurs upon the user's request forthe 5th web page, and the 4 page views that occurred prior to therequest for the 5th web page are considered to have occurred prior tothe conversion.

To track conversions (and other user interactions with the advertiser'swebsite), an advertiser can include, in the advertiser's web pages, aportion of code that monitors user interactions (e.g., page selections,content item selections, and other interactions) with advertiser'swebsite, and can detect a user interaction (or series of userinteractions) that constitutes a conversion. In some implementations,when a user accesses a web page, or another resource, from a referringweb page (or other resource), the referring web page (or other resource)for that interaction can be identified, for example, by execution of asnippet of code that is associated with the web page that is beingaccessed and/or based on a uniform resource locator (“URL”) that is usedto access the web page.

For example, a user can access an advertiser's website by selecting alink presented on a web page, for example, as part of a promotionaloffer by an affiliate of the advertiser. This link can be associatedwith a URL that includes data (i.e., text) that uniquely identifies theresource from which the use is navigating. For example, the linkhttp://www.example.com/homepage/%affiliate_identifier%promotion_(—)1specifies that the user navigated to the example.com web page from a webpage of the affiliate that is associated with the affiliate identifiernumber that is specified in the URL, and that the user was directed tothe example.com web page based on a selection of the link that isincluded in the promotional offer that is associated withpromotion_(—)1. The user interaction data for this interaction (i.e.,the selection of the link) can be stored in a database and used, asdescribed below, to facilitate performance reporting. When a conversionis detected for an advertiser, conversion data representing theconversion can be transmitted to a data processing apparatus (“analyticsapparatus”) that receives the conversion data, and in turn, stores theconversion data in a data store. This conversion data can be stored inassociation with one or more user identifiers (or other trackingmechanisms) for the user device that was used to perform the userinteraction, such that user interaction data associated with the useridentifier can be associated with the conversion and used to generate aperformance report for the conversion.

As used throughout this document, user interactions include anypresentation of content to a user and any subsequent affirmative actionsor non-actions (collectively referred to as “actions” unless otherwisespecified) that a user takes in response to presentation of content tothe user (e.g., selections of the content following presentation of thecontent, or no selections of the content following the presentation ofthe content). Thus, a user interaction does not necessarily require aselection (i.e., click) of the content (or any other affirmative action)by the user.

User interaction measures can include one or more of time lag measures(i.e., measures of time from one or more specified user interactions toa conversion), path length measures (i.e., quantities of userinteractions that occurred prior to conversions), user interaction paths(i.e., sequences of user interactions that occurred prior to theconversion), assist interaction measures (i.e., quantities of particularuser interactions that occurred prior to the conversion), assistedconversion measures (i.e., quantities of conversions that were assistedby specified content), and other measures, as described in more detailthroughout this document. Content providers can use the user interactionmeasures to adjust settings of a content distribution account thatcontrol allocation of their content, as described in more detail below

The description that follows generally describes user interactionmeasures as being provided to advertisers in the form of a report, wherethe user interactions measure user interactions with keyword targetedadvertisements prior to a sale by the advertiser. User interactionmeasures (and other performance measures) can also be provided for othercontent items (e.g., audio items, video items, games, and othermultimedia content) that are distributed using other distributioncriteria. The reports described below are generally described as onlinereports that are generated by a performance analysis apparatus that iscoupled to an advertisement management system. However, the userinteraction measures can be provided in other forms (e.g., distributedin various file formats and/or distributed in printed formats) and theperformance analysis apparatus can be implemented as an independentapparatus that is accessed, for example, over a data network.

FIG. 1 is a block diagram of an example environment 100 in which anadvertisement management system 110 manages advertising services. Theexample environment 100 includes a network 102, such as a local areanetwork (LAN), a wide area network (WAN), the Internet, or a combinationthereof. The network 102 connects websites 104, user devices 106,advertisers 108, and the advertisement management system 110. Theexample environment 100 may include many thousands of websites 104, userdevices 106, and advertisers 108.

A website 104 is one or more resources 105 associated with a domain nameand hosted by one or more servers. An example website is a collection ofweb pages formatted in hypertext markup language (HTML) that can containtext, images, multimedia content, and programming elements, such asscripts. Each website 104 is maintained by a publisher, which is anentity that controls, manages and/or owns the website 104.

A resource 105 is any data that can be provided over the network 102. Aresource 105 is identified by a resource address that is associated withthe resource 105. Resources include HTML pages, word processingdocuments, and portable document format (PDF) documents, images, video,and feed sources, to name only a few. The resources can include content,such as words, phrases, images and sounds, that may include embeddedinformation (such as meta-information in hyperlinks) and/or embeddedinstructions (such as JavaScript scripts).

A user device 106 is an electronic device that is under control of auser and is capable of requesting and receiving resources over thenetwork 102. Example user devices 106 include personal computers, mobilecommunication devices, and other devices that can send and receive dataover the network 102. A user device 106 typically includes a userapplication, such as a web browser, to facilitate the sending andreceiving of data over the network 102.

A user device 106 can request resources 105 from a website 104. In turn,data representing the resource 105 can be provided to the user device106 for presentation by the user device 106. The data representing theresource 105 can also include data specifying a portion of the resourceor a portion of a user display (e.g., a presentation location of apop-up window or in a slot of a web page) in which advertisements can bepresented. These specified portions of the resource or user display arereferred to as advertisement slots.

To facilitate searching of these resources, the environment 100 caninclude a search system 112 that identifies the resources by crawlingand indexing the resources provided by the publishers on the websites104. Data about the resources can be indexed based on the resource withwhich the data are associated. The indexed and, optionally, cachedcopies of the resources are stored in a search index 114. Data that areassociated with a resource is data that represents content included inthe resource and/or metadata for the resource.

User devices 106 can submit search queries 116 to the search system 112over the network 102. In response, the search system 112 accesses thesearch index 114 to identify resources that are relevant to the searchquery 116. The search system 112 identifies the resources in the form ofsearch results 118 and returns the search results 118 to the userdevices 106 in search results pages. A search result 118 is datagenerated by the search system 112 that identifies a resource that isresponsive to a particular search query, and includes a link to theresource. An example search result 118 can include a web page title, asnippet of text or a portion of an image extracted from the web page,and the URL of the web page. Search results pages can also include oneor more advertisement slots in which advertisements can be presented.

A search result page can be sent with a request from the search system112 for the web browser of the user device 106 to set a HypertextTransfer Protocol (HTTP) cookie. A cookie can represent, for example, aparticular user device 106 and a particular web browser. For example,the search system 112 includes a server that replies to the query bysending the search results page in an HTTP response. This HTTP responseincludes instructions (e.g., a set cookie instruction) that cause thebrowser to store a cookie, e.g., instructions requesting the browser tostore a cookie for the site hosted by the server. If the browsersupports cookies and cookies are enabled, every subsequent page requestto the same server will include the cookie for that server. The cookiecan store a variety of data, including a unique or semi-uniqueidentifier. The identifier can be anonymized so that the privacy ofusers is protected. For example, the semi-unique identifiers can beassociated with users, but the actual identifying information of theusers is not stored in the cookie. Additionally, any identified userinteractions can be generalized (for example, generalized based on userdemographics) rather than associated with a particular user. Encryptionand obfuscation techniques can also be used to protect the privacy ofusers. Because HTTP is a stateless protocol, the use of cookies allowsan external service, such as the search system 112 or other system, totrack particular actions and status of a user over multiple sessions.

When a resource 105 or search results 118 are requested by a user device106, the advertisement management system 110 receives a request foradvertisements to be provided with the resource 105 or search results118. The request for advertisements can include characteristics of theadvertisement slots that are defined for the requested resource orsearch results page, and can be provided to the advertisement managementsystem 110.

For example, a reference (e.g., URL) to the resource for which theadvertisement slot is defined, a size of the advertisement slot, and/ormedia types that are available for presentation in the advertisementslot can be provided to the advertisement management system 110.Similarly, keywords (i.e., one or more words that are associated withcontent) associated with a requested resource (“resource keywords”) or asearch query 116 for which search results are requested can also beprovided to the advertisement management system 110 to facilitateidentification of advertisements that are relevant to the resource orsearch query 116.

Based on data included in the request for advertisements, theadvertisement management system 110 can select advertisements that areeligible to be provided in response to the request (“eligibleadvertisements”). For example, eligible advertisements can includeadvertisements having characteristics matching the characteristics ofadvertisement slots and that are identified as relevant to specifiedresource keywords or search queries 116. In some implementations,advertisements having targeting keywords that match the resourcekeywords or the search query 116 are selected as eligible advertisementsby the advertisement management system 110. When a targeting keywordmatches a search query 116 or a resource keyword and the eligibleadvertisement associated with that targeting keyword is presented to auser (i.e., receives an impression), the targeting keyword is consideredto have caused presentation of the eligible advertisement.

A targeting keyword can match a resource keyword or a search query 116by having the same textual content (“text”) as the resource keyword orsearch query 116. For example, an advertisement associated with thetargeting keyword “hockey” can be an eligible advertisement for anadvertisement request including the resource keyword “hockey.”Similarly, the advertisement can be selected as an eligibleadvertisement for an advertisement request including the search query“hockey.”

A targeting keyword can also match a resource keyword or a search query116 by having text that is identified as being relevant to a targetingkeyword or search query 116 despite having different text than thetargeting keyword. For example, an advertisement having the targetingkeyword “hockey” may also be selected as an eligible advertisement foran advertisement request including a resource keyword or search queryfor “sports” because hockey is a type of sport, and therefore, is likelyto be relevant to the term “sports.”

Advertisers can specify a level of match that is required betweentargeting keywords that are used to distribute their advertisements anda resource keyword and/or a search query 116. For example, an advertisercan specify a targeting keyword as a broad match keyword, a phrase matchkeyword, or an exact match keyword. Broad match keywords are keywordsthat are matched by phrases (i.e., one or more words) that include oneor more same words as the broad match keywords. Broad match keywords canalso be matched by phrases that include stubs, synonyms, variousalternative forms of the broad match keywords (e.g., past tense,gerunds, future tense), and/or terms that are relevant to a same topicor concept as the broad match targeting keyword (e.g., based on termclustering techniques). For example, the broad match keyword “sportsequipment,” may be matched by “tennis equip” because this phraseincludes the word “equipment,” which is included in the broad matchkeyword. In this example, the phrase “hockey” can also match “sportsequipment” because hockey is a type of sport, and therefore, is relevantto the term “sports.” Conversion path performance measures describedbelow can be computed and/or presented on a per targeting keyword matchtype basis.

Phrase match keywords are keywords that are matched by phrases thatinclude at least one same word as the phrase match keyword. The phrasesthat match a phrase match keyword can include additional words that arenot included in the phrase match keyword. For example, if “sportsequipment” is designated as a phrase match keyword, the phrases“sports,” “equipment,” “sports equipment” and “new sports equipment”will match the phrase match keyword.

Exact match keywords are keywords that are matched by phrases thatinclude the exact same words as the exact match keywords. For example,the exact match keyword “hockey sticks” is matched by the phrase “hockeysticks,” and is not matched by the phrase “durable hockey sticks”because “durable hockey sticks” includes the word “durable,” which isnot included in the exact match keyword “hockey sticks.”

The advertisement management system 110 can select the eligibleadvertisements that are provided for presentation in advertisement slotsof a resource or search results page based on results of an auction. Forexample, the advertisement management system 110 can receive bids fromadvertisers and allocate the advertisement slots to the highest biddersat the conclusion of the auction.

Targeting keywords and other data associated with the distribution ofadvertisements can be stored in an advertising data store 119. Theadvertising data store 119 is a data store that stores references toadvertisements, advertising campaign parameters that are used to controldistribution of the advertisements, data representing conditions underwhich the advertisement was selected for presentation to a user, anddata representing actions taken by users in response to presentation ofthe advertisement. For example, the advertising data store can storetargeting keywords, bids, and other criteria with which eligibleadvertisements are selected for presentation.

The advertising data store can also store user interaction datarepresenting user interactions with presented advertisements (or othercontent items). For example, when an advertisement is presented to theuser, data can be stored in the advertisement data store 119representing the advertisement impression. In some implementations, thedata are stored in response to a request for the advertisement that ispresented. For example, the ad request can include data specifying auser identifier for a user device (e.g., a cookie), such that dataidentifying the user device can be stored in association with data thatidentifies the advertisement(s) that were presented in response to therequest.

When a user selects (i.e., clicks) a presented advertisement, data arestored in the advertisement data store 119 representing the selection ofthe advertisement. In some implementations, the data are stored inresponse to a request for a web page that is linked to by theadvertisement. For example, the user selection of the advertisement caninitiate a request for presentation of a web page that is provided by(or for) the advertiser. The request can include data specifying a useridentifier for the user device, and this data can be stored in theadvertisement data store.

The advertising data store 119 can also include data that specifies aquantity of impressions that each advertisement has received, and thequantity of impressions for each advertisement can be delineated, forexample, using the keywords that caused the advertisement to receiveimpressions. Each impression and user selection can be associated with(i.e., stored with references to and/or indexed according to) theadvertisement that was selected and/or the targeting keyword that causedthe advertisement to be selected for presentation. Similarly, eachimpression and user selection can be associated with an identifier(e.g., cookie identifier) for the device that received the impressionand from which the selection was made.

As described above, advertisers 108 can submit, to the advertisementmanagement system 110, campaign parameters (e.g., targeting keywords andcorresponding bids) that are used to control distribution ofadvertisements. Campaign parameters are parameters of a contentdistribution campaign that are used to control content distribution inresponse to content requests. For example, campaign parameters caninclude targeting keywords and corresponding bids, geographic ordemographic targeting criteria, as well as other parameterscorresponding to a set of advertisements.

A campaign is a set of one or more advertisements and correspondingcampaign parameters that are grouped together into a same advertisingunit. For example, advertisements for sporting equipment can be groupedtogether into a campaign. Within a campaign, subsets of theadvertisements can be grouped into “ad groups.” For example, an ad groupin the above-referenced sports equipment campaign can include a set ofadvertisements for baseball equipment.

The campaign parameters for each ad group can be referred to as ad groupparameters and, because ad group parameters are a subset of campaignparameters, changes to ad group parameters constitute changes tocampaign parameters. The campaign parameters corresponding to eachadvertisement can be referred to as advertisement parameters.Advertisers can also adjust advertisement parameters that are used tocontrol distribution of individual advertisements. Adjustment ofadvertisement parameters constitutes an adjustment of ad groupparameters and campaign parameters of the ad group(s) and campaign(s) towhich the advertisement belongs.

The advertisers 108 can access to the advertisement management system110 to monitor performance of the advertisements that are distributedusing the campaign parameters. For example, an advertiser can access acampaign performance report that provides a quantity of impressions(i.e., presentations), selections (i.e., clicks), and conversions thathave been identified for the advertisements. The campaign performancereport can also provide a total cost, a cost-per-click, and other costmeasures for the advertisement over a specified period of time.

For example, an advertiser may access a performance report thatspecifies that advertisements distributed using the phrase match keyword“hockey” have received 1,000 impressions (i.e., have been presented tousers 1,000 times), 20 clicks, and 5 conversions. Thus, the phrase matchkeyword hockey can be attributed with 1,000 impressions, 20 clicks, and5 conversions.

Typically a conversion is attributed to a targeting keyword when anadvertisement that is targeted using the targeted keyword is the “lastclicked advertisement” prior to the conversion. For example, advertiserX may associate the phrase match keywords “tennis,” “shoes,” and“Brand-X” with advertisements. In this example, assume that a usersubmits the search query “tennis,” the user is presented a search resultpage that includes advertiser X's advertisement, and the user selectsthe advertisement, but the user does not take an action that constitutesa conversion. Assume further that the user subsequently submits thesearch query “Brand-X,” is presented with the advertiser X'sadvertisement, the user selects advertiser X's advertisement, and theuser takes action that constitutes a conversion (e.g., the userpurchases Brand-X tennis shoes). In this example, the keyword “Brand-X”will be attributed the conversion because the last advertisementselected prior to the conversion (i.e., the “last selectedadvertisement”) was an advertisement that was presented in response tothe “Brand-X” being matched (i.e., the “Brand-X” keyword causedpresentation of the last clicked advertisement).

Attributing a conversion to the keyword that caused presentation of thelast selected advertisement (“last selection credit”) prior to aconversion is a useful measure of advertisement performance, but thismeasure alone does not provide advertisers with data that facilitatesanalysis of a conversion cycle that includes user exposure to, and/orselection of, advertisements prior to the last selected advertisement.For example, last selection credit measures alone do not specifykeywords that may have increased brand or product awareness throughpresentation of advertisements that were presented to, and/or selectedby, users prior to selection of the last selected advertisement. Thesepreviously presented and/or selected advertisements may have contributedsignificantly to the user subsequently taking action that constituted aconversion, but without data that provides user interaction measureswith advertisements prior to the last click, the contributions of thesekeywords to the conversion can be overlooked.

A conversion cycle is a period that begins with an initial userinteraction, by a particular user, with content provided for (or by) aparticular content provider (e.g., advertiser) and ends at a time atwhich the particular user takes an action that constitutes a conversionfor the particular content provider. The conversion cycle can bemeasured and/or constrained by time or actions and can span multipleuser sessions. For example, a conversion cycle can include a periodmeasured from a time at which a particular user was initially presentedan advertiser's advertisement to a time at which the user took an actionthat the advertiser has specified as a conversion (i.e., an action thatconstituted a conversion for that advertiser). Alternatively, aconversion cycle can be limited to a specified period of time precedinga conversion, such that user interactions prior to the specified periodof time are not considered part of the conversion cycle.

Conversion cycles for two different conversions can overlap, such thatone or more of the same user interactions can be associated with each ofthe conversions. For example, assume that a set of user interactions areassociated with a particular conversion (e.g., creating a user account).Further assume that shortly following the particular conversion, theuser performs another interaction (e.g., makes a purchase) thatconstitutes a second conversion. In this example, the user interactionsthat are associated with the first conversion can also be associatedwith the second conversion.

Each user session is a set of user interactions for a particular userthat are grouped together for analysis. The user sessions can includeuser interactions that occurred within a threshold period of time (e.g.,within 1 hour, 1 day, or 1 month) or can be delineated using specifiedactions. For example, a user search session can include user searchqueries and subsequent actions that occur over a 1 hour period and/oroccur prior to a session ending event (e.g., closing of a web browser).

Returning to the example above, the keyword “tennis” is not attributedcredit for the conversion, even though the advertisement that waspresented in response to a search query matching the keyword “tennis”may have contributed to the user taking an action that constituted aconversion (e.g., making a purchase of Brand-X tennis shoes). Forinstance, upon user selection of the advertisement that was presented inresponse to the keyword “tennis” being matched, the user may have viewedBrand-X tennis shoes that were available from advertiser X. Based on theuser's exposure to the Brand-X tennis shoes, the user may havesubsequently submitted the search query “Brand-X” to find the tennisshoes from Brand-X. Similarly, the user's exposure to the advertisementthat was targeted using the keyword “tennis,” irrespective of the user'sselection of the advertisement, may have also contributed to the usersubsequently taking action that constituted a conversion (e.g.,purchasing a product from advertiser X).

Analysis of user interactions, with an advertiser's advertisements (orother content provided by a content provider), that occur prior toselection of the last selected advertisement can enhance an advertiser'sability to understand the advertiser's conversion cycle and how itscustomers interact with advertisements over a conversion cycle. Forexample, if an advertiser is provided data specifying that, on average,the time from a converting user's (i.e., a user that takes an actionthat constitutes a conversion) first exposure to an advertisement to aconversion is 20 days, the advertiser can use this data to infer anamount of time that users spend researching alternative sources prior toconverting (i.e., taking actions that constitute a conversion).Similarly, if an advertiser is provided data specifying that aparticular targeting keyword causes presentation of a substantialportion of the advertisements that are initially presented to many ofthe converting users, the advertiser may decide to increase the amountof money that it spends on advertisements distributed using thatparticular keyword and/or increase the quality of advertisements thatare targeted using that particular keyword.

User interaction measures that facilitate analysis of user interactionsover a conversion cycle are referred to as conversion path performancemeasures. Conversion path performance measures specify durations ofconversion cycles and/or paths, quantities of user interactions thatoccurred during conversion cycles and/or paths, user interaction pathsthat preceded a conversion, as well as other measures of userinteraction that occurred during conversion cycles, as described in moredetail below.

The advertisement management system 110 includes a performance analysisapparatus 120 that determines conversion path performance measures. Theperformance analysis apparatus 120 obtains, for each advertiser, userinteraction data that represents user interactions with advertisementsthat are provided by the advertiser, determines (i.e., computes) one ormore conversion path performance measures, and provides data that causepresentation of a performance report specifying at least one of theconversion path performance measures. Using the performance report theadvertiser can analyze its conversion cycle and/or conversion paths, howusers interact with advertisements that are distributed using targetingkeywords irrespective of whether the keywords caused presentation of thelast selected advertisement before a conversion. In turn, the advertiseruses the performance report to make adjustments to campaign parametersthat control distribution of its advertisements.

FIG. 2 is a flow chart of an example process 200 for providing aconversion performance report. A conversion performance report caninclude one or more conversion path performance measures for conversionsthat occurred over a reporting period. For example, a conversionperformance report can include conversion path performance measures suchas time lag measures, path length measures, assist user interactions,and other performance measures for the conversions that occurred overthe previous 30 days. These conversion path performance measures can beprovided, for example, in a user interface similar to those describedwith reference to FIGS. 3B and 3C.

The process 200 is a process by which a request for a conversionperformance report is received and user interaction data that are usedto generate conversion path performance measures for the report areobtained. Using the user interaction data, conversions that occurredduring a report period are selected, and the user interaction dataassociated with these conversions are used to compute conversion pathperformance measures for the conversions. Once the conversion pathperformance measures are computed, the conversion performance report canbe populated with the conversion path performance measures. In turn,data that cause presentation of the conversion performance report areprovided to a user device.

The process 200 is described below with reference to advertisingcampaigns that control distribution of advertisements in an onlineenvironment. However, the process 200 can also be used with othercontent distribution campaigns that control distribution of othercontent (e.g., video, audio, or other content).

The process 200 can be implemented, for example, by the performanceanalysis apparatus 120 of FIG. 1. In some implementations, theperformance analysis apparatus 120 is a data processing apparatus thatincludes one or more processors that are configured to perform actionsof the process 200. In other implementations, a computer readable mediumcan include instructions that, when executed by a computer, cause thecomputer to perform actions of the process 200.

A request for a performance report is received (202). The request for aperformance report can be received, for example, from an advertiser forwhich advertisement distribution is being controlled by one or moretargeting keywords. The request can include data specifying a reportperiod for which the report is being requested. The report period canspecify, for example, that the requested report is to be generated forconversions that occurred during the report period. For example, if thereport period specifies a period of the previous 30 days, then thereport that is generated in response to the request can be generated forconversions that occurred in the last 30 days.

User interaction data that specifies user interactions with contentitems are obtained (204). In some implementations, the user interactiondata that are obtained are user interaction data for conversions thatoccurred during the reporting period. The user interaction data includesdata that specifies a conversion and a time at which the conversionoccurred. Thus, the user interaction data can be examined to identifyconversions that occurred during the last 30 days.

Conversion path performance measures are computed using the userinteraction data (206). As described in detail below, various conversionpath performance measures can be performed. For example, top pathmeasures, assisted conversion measures, first click measures, last clickanalyses, path length measures, and combinations thereof can becomputed. Additional conversion path performance measures can also becomputed, as described below. In some implementations, conversion pathperformance measures are computed on a per-keyword basis, a per-ad groupbasis, a per-ad campaign basis, a per-anonymized user basis and/or aper-advertiser basis. Other delineations of conversion path performancemeasures are also possible. For example, conversion path performancemeasures can also be computed on a per-keyword match type basis (e.g.,broad, phrase, and/or exact match type).

Once the conversion path performance measures are computed, theconversion performance report is generated to include the conversionpath performance measures, and data that cause presentation of theconversion performance report are provided (208). The data can beprovided, for example, to a user device that uses the data to causepresentation of the conversion performance report on a display device.

The processes, apparatus, and computer products that can be used tocompute conversion path performance measures and provide conversionperformance reports are provided in more detail below. FIGS. 3A-8Bdescribe various conversion path performance measures that can becomputed. FIG. 9 describes computing and providing conversion pathperformance measures using multiple sources of user interactions andFIGS. 10A and 11 describe components and apparatus with which conversionpath performance measures can be computed and reported.

Time Lag

FIG. 3A is a flow chart of an example process 300 for determining timelags for advertisement conversions. A time lag is an amount of timebetween a reference user interaction and a conversion. The referenceinteraction the user can be, for example, a first impression, a firstclick, or another specified interaction from which a time to conversioncan be computed.

In general, time lags represent how long it takes a potential customer(e.g., a user executing a browser on a client device 106) to convertfollowing the reference user interaction. For example, a user may spendmore time researching alternative products or services as the cost ofthe product or service increases. Therefore, a time lag for higherpriced items may be longer than the time lag for lower priced items, asusers may spend more time researching higher priced item (e.g., aEuropean vacation or a home theatre system) than they spend researchinglower priced items (e.g., a T-shirt).

The process 300 is a process by which one or more conversions areselected from user interaction data that specify user interactions withcontent items, and a time at which each conversion occurred isdetermined. In turn, a reference user interaction is selected from theuser interaction data for each conversion and a time at which eachreference user interaction occurred is determined. A time lag is thencomputed for each of the conversions using the time at which theconversion occurred and the time at which the reference user interactionoccurred. Measures of time lag (“time lag measures”) can be presented,for example, in a time lag report.

The process 300 is described below with reference to advertisingcampaigns that control distribution of advertisements in an onlineenvironment. However, the process 300 can also be used with othercontent distribution campaigns that control distribution of othercontent (e.g., video, audio, or other content).

The process 300 can be implemented, for example, by the performanceanalysis apparatus 120 of FIG. 1. In some implementations, theperformance analysis apparatus 120 is a data processing apparatus thatincludes one or more processors that are configured to perform actionsof the process 300. In other implementations, a computer readable mediumcan include instructions that, when executed by a computer, cause thecomputer to perform actions of the process 300.

Different types of time lags can be determined by the process 300,including a “first impression” time lag, a “first click” time lag, and alast click time lag. As used throughout this document, the firstimpression time lag is an amount of time between the time at which auser is first presented a content item (“first impression time”) and thetime at which the user performs an action that constitutes a conversion(“conversion time”). The first impression time lag can be used tomeasure a conversion cycle. The first click time lag is an amount oftime between the time at which the user first selects (e.g., clicks) acontent item and the conversion time. The last click time lag is anamount of time between the time at which the “last click” occurred andthe conversion time.

In some implementations, time lags are unbounded, such that any value oftime lag can be provided to an advertiser. In other implementations,time lags can have upper bounds that limit the maximum time lag that iscomputed and/or reported.

For example, the first impression time lag can have an upper bound of 30days. The upper bound can be determined based on data storageconstraints (e.g., administratively imposed) that limit the amount ofuser interaction data that are stored. For example, if data storageconstraints specify that 90 days of user interaction data are to bestored for each conversion, then the performance analysis apparatus 120can be configured to compute and/or provide time lags that are up to 90days long. Similarly, the advertising data store 119 can be configuredto maintain user interaction data that occurred within the upper bound.For example, the advertising data store can be configured such that userinteraction data that occurred outside of the upper bound expires and/oris overwritten by new user interaction data. The upper bounds used fortime lags can be established by the advertisement management system 110,or an administrator thereof, for use by the performance analysisapparatus 120.

In some implementations, the system 100 can use different upper boundsfor different advertisers 108. For example, some advertisers may offerproducts or services that have a longer average time lag than productsor services that are provided by other advertisers. Therefore, theadvertisers offering products or services that have longer average timelag may require a larger upper bound to receive accurate measures oftime lag than the upper bound required to provide the other advertiserswith accurate measures of time lag.

User interaction data specifying user interactions with content items isobtained (302). The content items can be, for example, ads presentedwith search results, banner ads, text ads, video content, or audiocontent. Forms of user interactions can include presentation of acontent item (i.e., an impression) to a user, selection (i.e., a click)of a content item by a user, and presentation of all or a portion of anaudio and/or video content item to the user. In some implementations,the user interaction data for each converting user (i.e., a user thatperforms an action that constitutes a conversion) are collected over aspecified time period (e.g., as specified by the upper bound). Forexample, user interaction data can be collected for all userinteractions that have occurred with each converting user over the lasttwo months (or other specified time periods).

As an example, referring to FIG. 1, the system 100 can obtain userinteraction data corresponding to advertisement conversions (e.g.,conversions related to “Brand X” tennis shoes) that are associated withuser interactions with content items using the client device 106. Theuser interactions can include queries 116 that were submitted by aconverting user, presentation of advertisements and search results 118that are responsive to the converting user's queries 116, and subsequentactions performed by the converting user after presentation of theadvertisements and search results 118. The advertisements that arepresented with the search results 118 can be advertisements that areassociated with one or more targeting keywords (“tennis,” “shoes,” and“Brand-X”), and were selected for presentation, for example, based onthe one or more targeting keywords being matched by a search query 116.If the user subsequently performs an act constituting a conversion(e.g., purchases “Brand-X” tennis shoes), the user interaction data canbe associated with the conversion. In some implementations, userinteractions for each conversion are one or more presentations ofcontent items during a conversion cycle for the conversion and zero ormore user selections of the content items that occurred during theconversion cycle. In other implementations, the user interactions can beone or more user selections of the content items during the conversioncycle (i.e., a specified period preceding the conversion).

The obtained user interaction data specify user interactions that areassociated with a particular advertiser. User interactions that areassociated with a particular advertiser are user interactions withcontent items provided by the advertiser. User interaction data thatrepresent user interactions are associated with the advertiser by beingstored with an advertiser identifier (e.g., an advertiser name, campaignname, and/or another identifier) that represents the advertiser and/orby being indexed according to the advertiser identifier or to a sameindex key (e.g., a same conversion identifier) as the ad campaignidentifier.

User interactions that are specified by the user interaction data canalso be associated with one or more conversions. For example, a userwith which one or more user interactions occurred (i.e., a user that waspresented one or more advertisements and/or selected one or more of thepresented advertisements) can subsequently purchase goods from aretailer (i.e., complete a conversion). In this example, the impressionsand/or selections of content items (e.g., advertisements) that occurredprior to the conversion can be identified as user interactions that areassociated with the conversion, and can be stored with the conversionidentifier that represents the conversion and/or be indexed according tothe conversion identifier. If another conversion occurs prior to anyadditional user interactions and/or within the conversion cycle for theother conversion, the user interactions described above can also beassociated with the other conversion in a similar manner.

For example, a user can be presented a first ad for a pizza chain andlater be presented a second ad for a pizza chain without selectingeither of the two ads. The user can then submit the URL for a web pageof the pizza chain and order a pizza online (i.e., perform an act thatconstitutes a conversion). In this example, the impressions (i.e., thepresentations of the two ads by the user) can be associated with theconversion (i.e., the purchasing of the pizza by the user), as describedabove. If the user again submits the URL of the web page and ordersanother pizza (or performs another act that constitutes a conversion)the impressions can also be associated with a second conversion,assuming that the impressions occurred within the conversion cycle for(i.e., within a specified period preceding) the second order.

As described above, each conversion can also be associated with contentitems that were presented and/or selected during the conversion cyclefor the conversion as well as targeting keywords that controldistribution of the content items. In some implementations, dataspecifying a targeting keyword that was matched, and therefore, caused acontent item to be presented during the conversion cycle for theconversion can be stored at a memory location associated with theconversion and/or indexed according to a conversion identifierrepresenting the conversion. For example, if the targeting keyword“clothing” causes presentation of an ad to a user and the usersubsequently selects (i.e., clicks) the presented ad, the keyword“clothing” can be associated with the user selection. If the usersubsequently completes a conversion, then the selection of the ad isidentified as a user interaction for the conversion.

User interactions and/or conversions can be associated with one or moread groups and/or ad campaigns based on the associations of the userinteractions and/or conversions with targeting keywords that belong tothe respective ad groups and/or ad campaigns. For example, a userinteraction that is associated with a particular targeting keyword canbe associated with ad groups and/or ad campaigns to which the particulartargeting keyword belongs based on the association of the userinteraction with the targeting keyword.

A conversion is selected from the user interaction data (304). In someimplementations, the conversion can be selected in response to receivinga request, from an advertiser, for a time lag report (i.e., a reportthat provides measures of time lag) for a content distribution campaign(e.g., ad campaign). For example, an advertiser may access a userinterface similar to that described with reference to FIGS. 3B and 3C,and request a time lag report that specifies one or more measures oftime lag for a particular ad campaign, particular ad group, orparticular targeting keyword. In turn, data for the conversions thatwill be used for computing the measures of time lag can be selected.

In other implementations, the conversion can be selected as part of aperiodic time lag report update. For example, a time lag report can begenerated and, in turn, updated periodically to provide an advertiserwith an updated time lag report upon demand. Pre-computing time lagreports can provide faster access to the reports in response to therequest. The description that follows describes selecting an individualconversion, but groups of conversions can also be selected forprocessing.

In some implementations, the performance analysis apparatus 120 selectsa conversion (e.g., a conversion related to Brand-X tennis shoes) from agroup of conversions that are stored in the advertising data store 119.The group can include the selected conversion related to Brand-X tennisshoes and other unrelated conversions, such as for pet toys or Europeanvacations.

For example, the performance analysis apparatus 120 can access theadvertising data store and identify each conversion that is associatedwith a particular advertiser for which a time lag report is beinggenerated or updated. The performance analysis apparatus 120 canidentify conversions that are associated with a particular advertiser,for example, by searching the user interaction data for the advertiseridentifier representing the particular advertiser and identifying theconversions to which each instance of the advertiser identifier isindexed, as described in more detail with reference to FIG. 10.

Once the conversions that are associated with the particular advertiserhave been identified, the performance analysis apparatus 120 can selectone or more of the conversions. The conversions can be selectedrandomly, in an order in which they are indexed in the user interactiondata, or in another predetermined order. For example, the userinteraction data can include a set of conversions that each has a uniqueconversion identifier that uniquely identifies the conversion with whichthe identifier is associated. In this example, the conversions can beselected in sequential order of the conversion identifiers. Eachconversion can also have a timestamp that specifies a time at which theconversion occurred, and the timestamp can be used to select conversionsin chronological order in which the conversions occurred.

A time at which the conversion occurred is determined (306). Asdescribed above, each conversion can have a timestamp that represents atime at which the conversion occurred. In some implementations, the timeat which the conversion occurred is the time at which the act thatconstituted the conversion was detected. For example, when theadvertisement management system 110 receives data specifying that aconversion occurred, the data can include a timestamp or theadvertisement management system can append a timestamp to the dataspecifying the occurrence of the conversion. In turn, the advertisementmanagement system 110 can store the data specifying the occurrence ofthe conversion and its corresponding timestamp in the advertising datastore 119. Therefore, when the performance analysis apparatus 120obtains the conversion from the advertising data store 119, theperformance analysis apparatus 120 can read the timestamp for theconversion to determine the time at which the conversion occurred.

A reference user interaction associated with the conversion is selected(308). In some implementations, the reference user interaction is a userinteraction that occurred prior to user selection (i.e., a user click)of a last selected content item (i.e., a last clicked content itempreceding the conversion) for the advertiser. Continuing with theBrand-X tennis shoes example, assuming that at least one advertisementwas presented to the advertiser before the last selected advertisement,the performance analysis apparatus 120 can select, as the reference userinteraction, the first presentation of an advertisement, such as anadvertisement for a new style of Brand X shoes, to the user.

The reference user interaction can be selected from the user interactiondata that are associated with the selected conversion. As describedabove, user interaction that is associated with each conversion can bestored with the conversion identifier and/or indexed according to theconversion identifier. Therefore, the user interaction data can besearched and/or filtered to identify the user interactions that areassociated with the conversion identifier for the conversion. In turn,the user interaction data can be analyzed to select the user interactiondata that represents the reference user interaction.

For example, when the reference user interaction is an initialpresentation of a content item prior to the presentation of the lastuser selected content item, the user interaction data can be analyzed toselect, as the reference user interaction, the initial (i.e., first)presentation of a content item to the user. The initial presentation ofa content item can be identified, for example, by analyzing timestampsfor the user interactions to identify the content item presentationhaving the earliest timestamp. Alternatively, when the user interactionsare stored in sequential order of occurrence, the first content itempresentation listed in the sequence can be selected as the referenceuser interaction.

A time at which the reference user interaction occurred is determined(310). As described above, the data representing each user interactioncan be stored with a timestamp representing a time at which the userinteraction occurred. Therefore, the time at which the reference userinteraction occurred can be determined by analyzing the timestamp forthe reference user interaction. As described above, the timestamp forthe reference user interaction can be appended to and/or stored with thedata representing the particular user interaction.

A determination is made whether more conversions remain to be processed(312). When more conversions remain to be processed, then the process300 selects another conversion (302). For example, the performanceanalysis apparatus 120 can determine that additional conversions remainto be processed, such as conversions that are being analyzed for a timelag report.

When no conversions remain to be processed, then measures of time lagare determined (314). The measures of time lag are measures of timebetween one or more reference user interactions and the conversions withwhich the user interactions are associated. The measures of time lagscan include, for example, total time lags for each conversion, anaverage time lag for all, or a subset, of the conversions, a maximumand/or minimum time lag for any conversion, a median time lag, or otherstatistical measures of the measure of time between the reference userinteractions and the associated conversions (e.g., standard deviation ormode).

In some implementations, the measures of time lag can be used togenerate a time lag distribution for the conversions. The time lagdistribution can be represented by a graphical histogram that organizesthe conversions according to their respective time lags and generates ahistogram the specifies a quantity of conversions that have time lagsthat are within each of multiple histogram bins, as described in moredetail with reference to FIG. 3C.

In some implementations, advertisers can provide data that specifies themonetary value of each conversion, or another measure of value for eachof the conversions. In these implementations, the measures of time lagcan include measures of value that are computed relative to the timelag. For example, the total value of all conversions that have a timelag between X and Y can be computed, and compared to the total value ofall conversions having a time lag between A and B. Similarly, averagevalues of these groups of conversions can be computed and compared toeach other to determine relationships between time lag and conversionvalues.

In some implementations, the measures of time lag can be computed on aper-targeting keyword basis. For example, the performance analysisapparatus 120, when determining advertisement conversion measures (e.g.,time lags for an advertiser) can categorize and summarize the measuresbased on keywords (e.g., “tennis,” “shoes,” and “Brand-X”),advertisement groups (e.g., a group of sporting equipmentadvertisements), and one or more advertisement campaigns (e.g., a groupof advertisements for tennis shoes).

Data are provided that cause presentation of the measures of time lag(316). In some implementations, For example, described in detail withreference to FIGS. 3B and 3C below, the performance analysis apparatus120 can provide data that causes presentation of the measures of timelag in a time lag report that is presented to the user in a userinterface. The data can be provided, for example, over the network 102of FIG. 1, and displayed to an advertiser, for example, with a computerdevice executing at the advertiser's location.

FIGS. 3B-3C are screen shots of example user interfaces 320 and 370 thatdisplay information for time lags associated with advertisementconversions. For example, an advertiser 108 (e.g., the advertiser ofBrand X shoes) can use the user interfaces 320 and 370 to display andreview information for advertisement conversion measures and reportsassociated with conversion time lags. In some implementations, the userinterfaces 320 and 370 are displayed in a web browser by an advertiser108 based on information provided over the network 102. In someimplementations, the information used in the user interfaces 320 and 370is based on conversion information determined by the advertisementmanagement system 110 and/or the performance analysis apparatus 120. Forexample, some conversion information displayed in the user interfaces320 and 370 can be determined using the process 300 described withreference to FIG. 3A.

The user interface 320 includes a header 321 and a data page 322. Theheader 321 (e.g., “MyAdvertisingAccount—Time Lag”) identifies theinformation in the user interface 320 as relating to conversion time laginformation associated with an advertising account. The data page 322contains the advertisement conversion information, specificallydisplaying and summarizing conversion time lag information for aspecific time period based on settings established by the user (e.g.,the advertiser 108 displaying the user interface 320).

The data page 322 includes a page title 323 (e.g., “Time Lag”) and adate range selection control 324. The page title 323 identifies the datapage 322 as containing “Time Lag” information, which is consistent withthe subject displayed in the header 321. The date range selectioncontrol 324 provides a way for the user (e.g., advertiser) to identify atime period for which to display information on the data page 322. Asshown, a current selection that the user has selected using the daterange selection control 324 is the month-long period from Mar. 16 toApr. 15, 2010. This time period can correspond to any time durationselected by the advertiser. For example, the user can select a timeperiod as short as a few hours up to a time period of several days orweeks, such as a 30-day duration. The value for the current selection(i.e., the month-long period) can typically be more useful to anadvertiser 108 if the time period includes conversions having a longerconversion cycle. This is because, in general, while some advertisementconversions can occur rapidly, even within a few seconds, others cantake several days.

In some implementation, when the user selects the date range selectioncontrol 324, a calendar interface (not shown) can appear. For example,the user can identify dates in the current selection by clicking onindividual cells on a calendar display, such as cells for Mar. 16 andApr. 15, 2010. In some implementations, the date range selection control324 can provide additional controls for specifying time ranges onspecific dates. An example range using date and time, for example, canbe noon on Mar. 16, 2010 to 4 PM on Mar. 17, 2010. In someimplementations, various controls (e.g., sliders, etc.) can be used toallow the user to specify dates and times, without having to type invalues explicitly. For example, a user can interact with drop downmenus, radio buttons, and/or other user interface elements to selectpre-selected or pre-specified values or ranges.

The data page 322 includes various settings for controlling the type ofinformation that is to be displayed on the data page 322 for the currenttime period selection. A time lag type control 326 allows the conversioninformation displayed to be based on different types of lag times. Inthis case, the time lag type control 326 provides user options todisplay conversion information from the “first impression,” from the“first click,” or from the “last click.” As shown in FIG. 3B, thecurrent selection 327 is set to display conversion information from thefirst click.

The time lag report 322 includes a conversion type selection control 328that enables the user to specify one or more types of conversions forwhich the time lag report includes related information. In someimplementations, the conversion type selection control 328 can be a dropdown menu having user selectable options. In other implementations, theconversion type selection control 328 can be a hypertext link or anotheruser interface control element (e.g., a text box, check box, or radiobutton). In the example shown, the conversion type selection control 328indicates that the time lag report includes information for “all” typesof conversions is presented in the time lag report 322.

Time lag reports can be generated for each of many different types ofconversions. In some implementations, each of the following user actionscan be identified as a different conversion type: signing up for amailing list, making a purchase, creating a user profile, making areservation, requesting presentation of a video, requesting presentationof audio, downloading one or more files, installing one or more programson the user device, or providing specified information.

Each advertiser can specify its own set of conversion types and the useractions that constitute a conversion for each conversion type. Forexample, an on-line gaming company can identify three types ofconversions. The first conversion type can be identified as aninformation conversion that is completed when a mailing address(physical or electronic) is received from a user. The second conversiontype can be identified as a download conversion that is completed when auser downloads and/or installs a trial version of an on-line game. Thethird conversion type can be identified as a purchase conversion that iscompleted when a user purchases an on-line game. In this example, theconversion type selection control 328 enables allows the user (i.e., theadvertiser) to request presentation of a time lag report that specifiesmeasures of time lags for purchase conversions and/or a time lag reportthat specifies measures of time lags for other types of conversions(e.g., information conversions or download conversions).

In some implementations, the time lag report 322 can separately specify,in a single user interface display, measures of time lags for more thanone conversion type. Continuing with the example above, in response to arequest for a time lag report that includes separate measures of timelags for purchase conversions and download conversions, data can beprovided that cause presentation of the measures of time lags for thepurchase conversions at a display location that is adjacent to thedisplay location of the measures of time lags for the downloadconversions. Thus, a user can directly compare the measures time lagsfor purchase conversions to the measures of time lags for the downloadconversions.

An analysis type selection control 330 allows the user to specify thetype of information displayed on the screen. The current setting for theanalysis type selection control 330 is “Time Lag”, which agrees with the“Time Lag” annotations indicated in the header 321 and the page title323. As an example of how to change the type of information displayed inthe user interface 320, the user can select a different setting (e.g.,“Path Length”) from the analysis type selection control 330. As aresult, path length-related information is displayed in user interface320 (as described below with reference to FIG. 4B), and the header 321and the page title 323 are updated to indicate the new selection (e.g.,“Path Length”).

A table portion 331 in the data page 322 provides the detailedinformation of interest within the user interface 320. For example, thetable portion 331 shown in FIG. 3B includes data values and displayareas related to conversion time lags. The table portion 331, asdepicted using the current settings for various user controls andsettings, includes rows of information arranged in three columns. A timelag ranges column 332 (e.g., “Time to conversion”) identifies ranges oftime lags corresponding to conversions. For instance, the time lagsranges listed in the time lag ranges column 332 include ranges for lessthan eight hours, 8-15 hours, and so on up to a range of 216 or morehours. Each of the time lag ranges identifies a group of conversions forwhich a total quantity of conversions is displayed in the table portion331.

A conversion counts column 334 identifies the quantity of conversionsthat occurred for each of the time lag ranges identified by the time lagranges column 332. A conversion percentages column 336 identifies theoverall percentage (e.g., relative to 100% of the conversions) that eachof the conversion ranges represents, on a row-by-row basis in the tableportion 331.

For example, a row 338 in the table portion 331 identifies 1,498 as thequantity of conversions that occurred within 24-47 hours of a firstclick (e.g., a user's first click on an ad impression). The value 1,498represents 29.96% of the overall total of conversions, as indicated bythe percentage displayed in the row 338.

In another example, as indicated in a row 340, a total of 748conversions occurred within 16-23 hours of the first click. This valuerepresents 14.96% of the conversions that occurred during the month-longperiod from Mar. 16 to Apr. 15, 2010.

In another example row 342, the value of 504 conversions, or 10.8%,occurred in less than eight hours. In other example rows 344 and 346, atotal of 21 conversions occurred within 192-215 hours, and a total of 53conversions occurred after 216 or more hours.

The user interface 320 includes a display type selection control 348that can be used, for example, to select one or more type of views fordisplaying information in the conversion percentages column 336. Forexample, the view selection 349 currently selected is the bar graphoption, as indicated by a bar graph icon. As a result, in addition tothe percentages displayed in the conversion percentages column 336, bargraphs of various lengths are also displayed, each proportional to thepercentage of conversions represented by a particular row (e.g., rows338, 340, 341, etc.). The longest bar graph in this case appears in therow 338, representing the highest quantity (e.g., 1498) of theconversions for any one conversion time lag ranges listed in the timelag ranges column 332.

The user interface 320 includes other controls and areas. For example,an address bar 350 can identify the address (e.g., URL) that isassociated with the current display of the user's browser, in this casetime lag information for an advertising account. An export control 352can provide the user with options for exporting information from theuser interface 320, primarily the information included in the table 331and the identification of any user settings used to generate the data. Ascroll bar 354 is provided in some implementations to allow the user toscroll to other parts of the user interface 320, for example if theentries in the table 331 are outside the viewport of the data page 322.

The data page 322 includes a summary 356 (e.g., “Most conversionsoccurred within 24-47 hours”) which can be used to direct the user'sattention to specific information on the user interface 320. Forexample, the summary 356 in this case points out that the highestquantity of conversions corresponds to the information displayed in therow 338.

As shown in FIG. 3C, the information displayed in the user interface 370differs from that displayed in the user interface 320. The data page 322in the user interface 370 includes a pie chart display 372 that is canappear, for example, as a result of a different view selection 349 thatis chosen by the user from the display type selection control 348. Thiscontrasts with the bar chart display on the data page 322 shown in FIG.3B.

The pie chart 372 includes sectors 374 a through 374 x, eachrepresenting a conversion percentage that appears in the correspondingrow in the table portion 331. For example, the percentage 10.08 thatappears in the row 342 is represented in the pie chart display 372 asthe sector 374 a. Continuing clockwise from the sector 374 a, the sector374 b (e.g., marked 8.44%) corresponds to the row 341, the sector 374 c(e.g., marked 14.96%) corresponds to the row 340, and so on through thesector 374 x (e.g., marked 1.06%) that corresponds to the row 346. Thelargest sector 374 d corresponds to the row 338 in the table portion 331that has the highest percentage, 29.96%, and includes a label 396indicating the percentage. Different shading of the individual sectors(e.g., sectors 374 a-374 x) in the pie chart 372 corresponds to theshading patterns shown in the shading key 378. In some implementations,shading patterns are replaced by color shading.

The user interface 370 also includes a view selection popup 380 thatincludes view options that can be available, for example, upon selectionof the analysis type selection control 330. As shown, a “Time Lag”selection 382 is currently selected from the view selection popup 380.As a result, the information displayed in the user interface 370 isinformation related to time lags before advertisement conversions.

Path Length

FIG. 4A is a flow chart of an example process 400 for determining pathlength for advertisement conversions. A path length is a measure of howmany user interactions are included in a conversion path for aconversion. A conversion path for a conversion can include eachimpression and/or click (or other content-related user interactions) ofa content item (ads) that is associated with a particular conversion bya particular user. Thus, the conversion path for a particular conversioncan include every user interaction during the conversion cycle for theconversion.

An example conversion path related to a conversion for Brand X tennisshoes can include five user interactions, such that the conversion has apath length of five. In this example, a user can first be presented withan ad for Brand X (i.e., an impression of the ad occurs). The ad (“Ad1”) can be a general ad for the Brand X manufacturer, for instance, andnot include a specific reference to shoes. This initial impression of Ad1 can represent the first user interaction in the user's path toconversion. If the user clicks on Ad 1, the act of clicking Ad 1 can beconsidered another user interaction in the conversion path. The user cansubsequently be presented with a Brand-X tennis equipment ad (“Ad 2”),which can represent a third user interaction in the conversion path. Afourth user interaction in the conversion path can be presentation of aBrand X tennis shoe ad (“Ad 3”) and the user's selection of Ad 3 can bethe “last click” and the final user interaction in the conversion pathprior to converting (e.g., buying a pair of Brand X tennis shoes).

In general, path lengths represent how many user interactions occurduring a conversion cycle for a conversion. For example, a user mayperform more searches (i.e., submit more search queries) researchingalternative products or services as the cost of the product or servicethat they are researching increases. Therefore, the quantity ofinteractions with the user may also increase, such that a path lengthfor higher priced items may be longer than the path length for lowerpriced items. The quantity of user interactions during a conversioncycle, and in turn, path lengths can also vary for reasons other thanthe price of the product or service being offered. For example, the pathlength may vary based on the quality of the advertisements that arepresented to the user and/or how well targeted the advertisements are.

The user interactions of a conversion path can be represented bykeywords that caused presentation of the presented and/or clickedcontent items, URLs of web pages with which the content items werepresented, and/or interaction indicators that specify whether thecontent item was presented and/or selected. Representations of the userinteractions can be presented, for example, according to a sequence inwhich the user interactions occurred. For example, the conversion pathassociated with the user's purchase of Brand X tennis shoes can beillustrated by a sequential listing of the keywords that causedpresentation of the content items.

The process 400 is a process by which one or more conversions areselected from user interaction data that specify user interactions withcontent items. In turn, a path length (i.e., a quantity of userinteractions in a conversion path) is then determined for each of theconversions using impressions and/or clicks that are associated with theconversion that are included in the conversion path for the conversionpath length measures are computed using the path lengths of theconversions, and the path length measures can be presented, for example,in a path length report.

The process 400 is described below with reference to advertisingcampaigns that control distribution of advertisements in an onlineenvironment. However, the process 400 can also be used with othercontent distribution campaigns that control distribution of othercontent (e.g., video, audio, games, or other content).

The process 400 can be implemented, for example, by the performanceanalysis apparatus 120 of FIG. 1. In some implementations, theperformance analysis apparatus 120 is a data processing apparatus thatincludes one or more processors that are configured to perform actionsof the process 400. In other implementations, a computer readable mediumcan include instructions that, when executed by a computer, cause thecomputer to perform actions of the process 400.

Path length measures can be determined for various types of userinteractions. For example, impression path length measures and clickpath length measures can be separately computed using the method 400. Asused throughout this document, the impression path length measure is apath length measure that is computed using the quantity of content itemimpressions (e.g., a total number of search ad impressions) thatoccurred during conversion cycles for selected conversions. In someimplementations, the quantity of content item impressions is determinedusing only content item impressions that did not result in a click. Inother implementations, the quantity of content item impressions isdetermined using a total quantity of content item impressions,irrespective of whether the impressions resulted in a click by the user.Click path length measures are measures of path length that are computedusing a quantity of content item clicks (i.e., user selections ofcontent items) that occurred during conversion cycles of selectedconversions. The quantity of content item clicks can be, for example, atotal quantity of search ad clicks that preceded a conversion. In someimplementations, the quantity of content item clicks for each conversioncan include the “last click” for the conversion while in otherimplementations, the last click can be omitted from the quantity ofcontent item clicks.

User interaction data specifying user interactions with content itemsare obtained (402). As described above, content items can be ads thatare presented with search results, banner ads, text ads, video content,or audio content and user interactions can include presentations ofcontent items, user selections (i.e., clicks) of content items, andpresentation of an audio and/or video content item. In someimplementations, the user interaction data includes data representingall user interactions prior to a conversion. In other implementations,the user interaction data includes data representing user interactionsthat occurred within a specified period of time of the conversion (e.g.,within three months of the conversion). As described in more detailbelow, the user interaction data that is obtained can include a set ofconversions for one or more advertiser, the user identifiersrepresenting converting users for each of the conversions, and datarepresenting user interactions with advertisements during the conversioncycle and by the converting users.

In some implementations, the obtained user interaction data specify userinteractions that are associated with a particular advertiser and/oruser interactions that are associated with one or more conversions forthe particular advertiser. User interactions that are associated with aparticular advertiser are user interactions with content items providedby (or provided for) the advertiser. User interaction data can beassociated with the advertiser by being stored with an advertiseridentifier (e.g., an advertiser name, campaign name, and/or anotheridentifier) that represents the advertiser and/or by being indexedaccording to the advertiser identifier or indexed to a same index key(e.g., a same conversion identifier) as the advertiser identifier.

As described above, user interactions that are specified by the userinteraction data can also be associated with one or more conversions.For example, impressions and/or selections (i.e., clicks) of contentitems (e.g., advertisements) that occurred prior to a conversion can beidentified as user interactions that are associated with the conversionand user interaction data representing the user interactions that areassociated with the conversion can be stored with a conversionidentifier that represents the conversion and/or by being indexedaccording to the conversion or indexed to a same index key as theconversion identifier. If another conversion occurs prior to anyadditional user interactions and/or within the conversion cycle for theother conversion, the impressions and/or clicks can also be associatedwith the other conversion in a similar manner.

As described above, each conversion can also be associated with contentitems that were presented to and/or selected by a particular user duringthe conversion cycle for the conversion as well as targeting keywordsthat control distribution of the content items. In some implementations,data specifying a targeting keyword that was matched, and therefore,caused a content item to be presented during the conversion cycle forthe conversion can be stored at a memory location associated with theconversion and/or indexed according to a conversion identifierrepresenting the conversion. User interactions and/or conversions can beassociated with one or more ad groups and/or ad campaigns based on theassociations of the user interactions and/or conversions with targetingkeywords that belong to the respective ad groups and/or ad campaigns, asdescribed above with reference to FIG. 3A.

A conversion is selected from the user interaction data (404). In someimplementations, the conversion can be selected in response to receivinga request, from an advertiser, for a path length report (i.e., a reportthat provides path length measures) for a content distribution campaign(e.g., ad campaign). For example, an advertiser may access a userinterface similar to that described with reference to FIG. 4B, andrequest a path length report that specifies one or more path lengthmeasures for a particular ad campaign, particular ad group, orparticular targeting keyword. In turn, conversions for computing thepath length measures can be selected.

In other implementations, the conversion can be selected as part of aperiodic path length report update. For example, a path length reportcan be generated and, in turn, updated periodically to provide anadvertiser with an updated path length report upon demand. Pre-computingpath length reports can provide faster access to the reports in responseto the request. The description that follows describes selecting anindividual conversion, but groups of conversions can also be selectedfor processing.

In some implementations, the performance analysis apparatus 120 selectsthe conversion (e.g., a conversion related to Brand-X tennis shoes). Forexample, the performance analysis apparatus 120 can access theadvertising data store 119 and identify each conversion that isassociated with a particular advertiser for which a path length reportis being generated or updated. The performance analysis apparatus 120can identify conversions that are associated with a particularadvertiser, for example, by searching the user interaction data for theadvertiser identifier representing the particular advertiser andidentifying the conversions by which each instance of the advertiseridentifier is indexed, as described in more detail with reference toFIG. 10.

Once the conversions that are associated with the particular advertiserhave been identified, the performance analysis apparatus 120 can selectone or more of the conversions. The conversions can be selectedrandomly, in an order in which they are indexed in the user interactiondata, or in another predetermined order. For example, the userinteraction data can include a set of conversions that each has a uniqueconversion identifier that uniquely identifies the conversion with whichthe identifier is associated. In this example, the conversions can beselected in sequential order of the conversion identifiers. Eachconversion can also have a timestamp that specifies a time at which theconversion occurred, and the timestamp can be used to select conversionsin chronological order in which the conversions occurred.

User interactions that are associated with the selected conversion areselected from the user interaction data (406). In some implementations,the selected user interactions are user interactions with the convertinguser prior to an action that constituted a conversion. When the userinteraction data are indexed according to the conversion with which theuser interaction data are associated, as described in more detail withreference to FIG. 10, the user interactions can be selected by examining(e.g., searching) and/or filtering the user interaction data that areindexed to the selected conversion.

In some implementations, the identified user interactions for each userare interactions for that user that occurred at any time prior to theconversion that is associated with that user. In other implementations,the identifier user interactions are user interactions that occurredwithin a specified time preceding the conversion. For example, theidentified user interactions may include the user interactions thatoccurred within 30 days preceding the conversion, but exclude userinteractions that occurred prior to the 30 days preceding theconversion.

The specified time can be specified by the advertiser or determined, forexample, on an advertiser specific basis according to an analysis of thetime lag measures for each advertiser. In some implementations, thespecified time for each advertiser can be selected as a threshold periodthat exceeds time lag measures for at least a minimum portion of theadvertiser's conversions. For example, using the time lag measures forthe advertiser, a specified time can be selected that is greater thanthe first impression time lag for at least 67% of the advertiser'sconversions.

For example, when the reference user interaction is an initialpresentation of a content item prior to the presentation of the lastclicked content item, the user interaction data can be analyzed toselect, as the reference user interaction, the initial (i.e., first)presentation of a content item to the user. The initial presentation ofa content item can be identified, for example, by analyzing timestampsfor the user interactions to identify the content item presentationhaving the earliest timestamp. Alternatively, when the user interactionsare stored in sequential order of occurrence, the first content itempresentation listed in the sequence can be selected as the referenceuser interaction.

Once the reference user interaction has been selected the userinteractions that occurred following the reference user interaction, butprior to the conversion, can be selected from the user interaction data.When a timestamp is stored with each user interaction, the selected userinteractions can be user interactions having timestamps that specify atime between the timestamp for the reference user interaction and thetimestamp for the conversion. When the user interactions are stored insequential order of occurrence, each of the user interactions that arelocated between the reference user interaction and the conversion can beselected.

A quantity of user interactions that are associated with the conversionis determined (408). In some implementations, the determination is madeusing the user interaction data for the selected user interactions thatoccurred during the conversion cycle for the conversion. For example,the quantity of user interactions can be determined using an impressionvolume (e.g., a quantity of impressions) and a click volume (e.g., aquantity of clicks) that includes the last click for the conversioncycle.

In other implementations, the determination is made using the userinteraction data for the selected user interactions that occurred duringthe conversion cycle for the conversion and prior to the last click forthe conversion. For example, the quantity of user interactions can bedetermined using an impression volume (e.g., a quantity of impressions)and a click volume (e.g., a quantity of clicks) that does not includethe last click for the conversion cycle. In some implementations, thequantity of user interactions associated with the selected conversioncan be determined by summing of a quantity of impressions that areassociated with the selected conversion and a quantity of userselections of content items for the conversion.

A determination is made whether more conversions remain to be processed(410). When more conversions remain to be processed, then the process400 selects another conversion (404). For example, when a path lengthreport is being generated for a particular advertiser, the performanceanalysis apparatus 120 can determine that additional conversions remainto be processed for the particular advertiser.

When no conversions remain to be processed, a path length measure iscomputed for the conversion (412). In some implementations, the pathlength measure can represent an aggregate quantity of user interactionsthat occurred during conversion cycles for a set of conversions. Oneexample of a path length measure is an average path length for theconversions. The average path length for the conversions can be computedas a ratio of the total quantity (i.e., the value of the sum) of contentitem impressions and user selections of content items that are includedin a conversion path for the conversion. For example, an average pathlength of four can represent that on average, four user interactionsoccur prior to a conversion. This example conversion path can include animpression of the Brand X ad, a subsequent user selection (i.e., click)of the Brand X ad, an impression of the Brand-X tennis equipment ad, andthe user's last click on the ad for Brand X tennis equipment ad.

Other path length measures can also be computed. For example, a maximumpath length for the conversions, minimum path length for theconversions, a median path length, and other statistical measures (e.g.,standard deviation) can be computed. Additionally, the quantity of userinteractions associated with each conversion can be used to generate ahistogram that represents the distribution of conversions according topath length, as described in more detail with reference to FIGS. 4B and4C.

Path length measures can be filtered and/or computed on a userinteraction type basis. In some implementations, the path length measureis computed by computing a click path length measure that specifies aquantity of advertisement selections, by converting users, that areincluded in the conversion path. For example, the click path lengthmeasure can be computed using the quantity of advertisements clicks thatare included in the conversion path for the conversion, but excludingthe quantity of impressions for the conversion path. In theseimplementations, the quantity of user interactions that are associatedwith the conversion can be determined to be a quantity of userselections of advertisements that are included in the conversion pathfor the selected conversion.

In some implementations, the path length measure can be computed bycomputing an impression path length measure for the conversion cycle.The impression path length measures the quantity of search queries thatwere submitted during the conversion cycles and matched the targetingkeywords. For example, converting users can submit several queries thatmatch targeting keywords prior to the conversion. The impression pathlength measure can be computed using the quantity of advertisementimpressions that are included in the conversion path for the conversion,but excluding the quantity of clicks for the conversion path. In theseimplementations, the quantity of user interactions that are associatedwith the conversion can be determined to be a quantity of advertisementimpressions that are included in the conversion path for the selectedconversion.

Data are provided that cause presentation of the path length measure(414). For example, as described in detail with reference to FIG. 4Bbelow, the performance analysis apparatus 120 can provide data thatcauses presentation of the path length measure in a path length reportthat is presented to the user in a user interface. The data can beprovided, for example, over the network 102 and displayed to anadvertiser, for example, with a computer device executing at theadvertiser's location.

In some implementations, the process 400 further includes obtainingvalue measures for the conversions. Each value measure is a measure ofmonetary value or another measure of value that is assigned to aparticular conversion. An aggregate value of the conversions is computedusing the value measures, and data are provided that causes presentationof the aggregate value of the conversions. In some implementations, theaggregate value can be a total value, an average value, or anothercomputed aggregate value that is representative of the values of theconversions.

For example, using conversions related to Brand X tennis shoes, theperformance analysis apparatus 120 can determine the aggregate dollarvalue of conversions related to tennis shoes bought from Brand X ads.The value can be, for example, an average that is computed based onseveral conversions (e.g., by the same user or several different usersover time). The aggregate value can be provided for presentation to theadvertiser, such as in a user interface described below with referenceto FIG. 4B.

In some implementations, a histogram of path length measures, such asthe histogram of FIG. 4B, can be generated, where each histogram grouprepresents a particular path length (or range of path lengths) andspecifies a quantity of conversions having a path length that matchesthe particular path length (or is within the range of path lengths). Inthese implementations, an aggregate value can be determined for theconversions that are represented by the histogram group. For example,ten conversions having a path length of three can be represented by ahistogram bar having a value of ten. In this example, the aggregatevalue for the ten conversions can be a total value of the tenconversions, a ratio of the total value relative to the quantity ofconversions (i.e., total value/no. of conversions), or anotherstatistical measure of aggregate value. The performance analysisapparatus 120 can generate the histograms and provide information forpresenting a histogram that represents the aggregate values associatedwith conversions.

FIG. 4B is a screen shot of an example user interface 420 for providingpath length measures for conversions. For example, an advertiser 108(e.g., the advertiser of Brand X shoes) can use the user interface 420to display and review information for advertisement conversion measuresand reports associated with conversion path lengths. In someimplementations, the user interface 420 is displayed in a web browser byan advertiser 108 based on information provided over the network 102. Insome implementations, the information provided in the user interface 420includes conversion information (e.g., path length measures forconversions and associated data) provided by the advertisementmanagement system 110 and/or the performance analysis apparatus 120. Forexample, some conversion information displayed in the user interface 420can be determined using the process 400 described with reference to FIG.4A.

The user interface 420 includes a header 421 and a path length report422. The header 421 (e.g., “MyAdvertisingAccount—Path length”)identifies the information in the user interface 420 as relating toconversion path length information associated with an advertisingaccount. The path length report 422 contains the advertisementconversion information, specifically displaying and summarizingconversion path length information for a specific time period based onsettings established by the user (e.g., the advertiser 108 displayingthe user interface 420).

The path length report 422 includes a page title 423 (e.g., “PathLength”) and a date range selection control 424. The page title 423identifies the path length report 422 as containing “Path Length”information, which is consistent with the subject displayed in theheader 421. The date range selection control 424 enables the user (e.g.,advertiser) to specify a date range for the assisted conversion measuresthat are presented in the path length report 422. The date rangespecified in the date range selection control 424 can be used to selectdata with which the path length measures presented in the path lengthreport 422 are computed. For example, the date range can be used toselect user interaction data that are associated with conversions thatoccurred within the specified date range, user interaction data that areassociated with conversion paths (or cycles) where the entire conversionpath occurred during the specified date range, or user interaction datathat are associated with conversion paths where any portion of theconversion path occurred during the specified date range.

As shown, a current time period selection 425 that has been selectedusing the date range selection control 424 is the month-long period fromMar. 16 to Apr. 15, 2010. This time period can correspond to any timeduration selected by the advertiser. For example, the user can select atime period as short as a few hours up to a time period of several daysor weeks, such as a 30-day duration, and years. Some advertisers mayselect a longer date range than others because, in general, while someadvertisement conversions can occur rapidly, even within a few seconds,others can take several days, weeks, or even years. As a result, pathlengths that originate, for example, from the first ad impression thatis presented the user, can span several hours or days before aconversion occurs.

In some implementations, when the user selects the date range selectioncontrol 424, a calendar interface (not shown) can appear. For example,the user can identify dates in the current selection 425 by clicking onindividual cells on a calendar display, such as cells for Mar. 16 andApr. 15, 2010. In some implementations, the date range selection control424 can provide additional controls for specifying time ranges onspecific dates. An example range using date and time, for example, canbe noon on Mar. 16, 2010 to 4 PM on Mar. 17, 2010. In someimplementations, various controls (e.g., sliders, etc.) can be used toallow the user to specify dates and times, without having to type invalues explicitly. For example, a user can interact with drop downmenus, radio buttons, and/or other user interface elements to selectpre-selected or pre-specified values or ranges.

The path length report 422 includes various settings for controlling thetype of path length report that is displayed in the user interface 420for the current time period selection 425. A path length type control426 can include, for example, active links or other user interfaceelements that upon selection cause submission of a request for one ormore different types of path length reports. In the present example, thepath length type control 426 can cause presentation of a click pathlength report or an impression path length report. For example, userselection of the “clicks” active link can cause submission of a requestfor a click path length report, while selection of the “impressions”active link can cause submission of a request for an impression pathlength report. In response to the submission, data that causespresentation of the selected report can be provided to the user device.As shown in FIG. 4B, the current selection 427 is set to display a clickpath length report and a click path length report is presented in theuser interface 420.

The path length report 422 includes a conversion type selection control428 that enables the user to specify one or more types of conversionsfor which the path length report includes related information. In someimplementations, the conversion type selection control 428 can be a dropdown menu having user selectable options. In other implementations, theconversion type selection control 428 can be a hypertext link or anotheruser interface control element (e.g., a text box, check box, or radiobutton). In the example shown, the conversion type selection control 428indicates that the path length report includes information for “all”types of conversions is presented in the path length report 422.

Path length reports can be generated for each of many different types ofconversions. In some implementations, each of the following user actionscan be identified as a different conversion type: signing up for amailing list, making a purchase, creating a user profile, making areservation, requesting presentation of a video, requesting presentationof audio, downloading one or more files, installing one or more programson the user device, or providing specified information, and a separatepath length report can be generated for each of these differentconversion types.

Each advertiser can specify its own set of conversion types and the useractions that constitute a conversion for each conversion type. Forexample, an on-line gaming company can identify three types ofconversions. The first conversion type can be identified as aninformation conversion that is completed when a mailing address(physical or electronic) is received from a user. The second conversiontype can be identified as a download conversion that is completed when auser downloads and/or installs a trial version of an on-line game. Thethird conversion type can be identified as a purchase conversion that iscompleted when a user purchases an on-line game. In this example, theconversion type selection control 428 enables allows the user (i.e., theadvertiser) to request presentation of a path length report thatspecifies path length measures for purchase conversions and/or an pathlength report that specifies path length measures for other types ofconversions (e.g., information conversions or download conversions).

In some implementations, the path length report 422 can separatelyspecify, in a single user interface display, path length measures formore than one conversion type. Continuing with the example above, inresponse to a request for a path length report that includes separatepath length measures for purchase conversions and download conversions,data can be provided that cause presentation of the path length measuresfor the purchase conversions at a display location that is adjacent tothe display location of the path length measures for the downloadconversions. Thus, a user can directly compare the path length measuresfor purchase conversions to the path length measures for the downloadconversions.

An analysis type selection control 430 allows the user to specify thetype of information displayed on the screen. The current setting for theanalysis type selection control 430 is “Path length”, which agrees withthe “Path length” annotations indicated in the header 421 and the pagetitle 423. As an example of how to change the type of informationdisplayed in the user interface 420, the user can select a differentsetting (e.g., “Time Lag”) from the analysis type selection control 430.In response to the change, information related to the different setting(e.g., time lags) is displayed (as described above with reference toFIG. 3B), and the header 421 and the page title 423 are updated toindicate the new selection (e.g., “Time Lag”).

A table portion 431 of the path length report 422 provides detailed pathlength information within the user interface 420. For example, the tableportion 431 shown in FIG. 4B includes data values and display areasrelated to conversion path lengths. The table portion 431, as depictedusing the current settings for various user controls and settings,includes rows of information arranged in three columns. A clicks column432 (e.g., “Clicks before conversion”) identifies ranges of path lengthscorresponding to conversions. For instance, the path lengths listed inthe clicks column 432 specify quantities of clicks that occur beforeconversion (i.e., click path lengths) ranging from fewer than 1 click upto 12 or more clicks.

A conversion counts column 434 identifies the quantity of conversionsthat occurred for each of the path lengths identified by the clickscolumn 432. A conversion percentages column 436 identifies the overallpercentage (e.g., relative to 100% of the conversions) that each of theconversion counts represents, on a row-by-row basis in the table portion431.

For example, a row 438 in the table portion 431 identifies 7200 as thequantity of conversions that occurred with three clicks. The value 7200represents 49.62% of the overall total of conversions, as indicated bythe percentage displayed in the row 438.

In another example, as indicated in a row 440, a total of 3522conversions occurred with two clicks. This value represents 24.05% ofthe conversions that occurred during the month-long period from Mar. 16to Apr. 15, 2010.

In another example row 442, the value of 190 conversions, or 1.3%,occurred in less than one click. In other example rows 444 and 446, atotal of 14 conversions occurred with 11 clicks, and a total of 56conversions occurred with 12 or more clicks.

The user interface 420 includes a display type selection control 448that can be used, for example, to select one or more types of views fordisplaying information in the conversion percentages column 436, orelsewhere in the user interface. For example, the view selection 449that is currently selected is the histogram view option, as indicated bya histogram icon. As a result, in addition to the percentages displayedin the conversion percentages column 436, path length measures are alsopresented in a histogram (i.e., in bar graph form), and a length of eachgroup (or bar) is proportional to the percentage of conversions that arerepresented by a particular row (e.g., rows 438, 440, 441, etc.). Thelargest group (or longest bar) in this case appears in the row 438,representing the highest quantity (e.g., 7,200) of the conversions forany one quantity of clicks in column 432.

The user interface 420 includes other controls and areas. For example,an address bar 450 can identify the address (e.g., URL) that isassociated with (e.g., identifies a network location for) the data thatcauses presentation of the current display of the user's browser, inthis case path length information for an advertising account. An exportcontrol 452 can provide the user with options for exporting informationfrom the user interface 420, primarily the information included in thetable 431 and the identification of any user settings used to generatethe data. The export control 452 can also be used to export userinteraction data and/or data that may have been computed as describedherein, but has not been presented. A scroll bar 454 is provided in someimplementations to allow the user to scroll to other parts of the userinterface 420, for example, if the entries in the table 431 are outsidethe viewport of the path length report 422.

The path length report 422 includes a summary 456 (e.g., “Mostconversions occurred with a 3 click path”) which can be used to directthe user's attention to specific information on the user interface 420.For example, the summary 456 in this case points out that the highestquantity of conversions corresponds to the information displayed in therow 438.

Top Paths

FIG. 5A is a flow chart of an example process 500 for selecting toppaths for advertisement conversions. Top paths are conversion paths thatare associated with at least a specified minimum quantity ofconversions. In some implementations, the top paths for an advertiserare a threshold number of conversion paths for the advertiser thatresulted in the highest quantities of conversions for the advertiser.Each conversion path in the top paths specifies an order (or path) ofoccurrence (i.e., an interaction sequence) for the set of userinteractions that occurred during a conversion cycle for the conversion.The conversion path further specifies keywords that caused presentationof content items that were presented to and/or selected by the user(i.e., clicked by the user). As a result, the top paths measure providesconversion measures for the highest conversion-producing conversionpaths, and identifies the keywords that are associated with the topconversion paths.

For example, a set of top paths for an advertiser can specify tenconversion paths that provided the ten highest total number ofconversions for the advertiser. Each of these top paths can specify aset of targeting keywords that caused presentation of advertisementswith which the user interacted, where the keywords are presentedaccording to the order in which the content items were presented to theuser. Each of the top paths can also specify, for each of the keywords,the type of interaction that occurred with the user. For example, aclick identifier can be presented with the keywords that causedpresentation of clicked advertisements in the conversion path, while animpression identifier can be presented with the keywords that causedpresentation of non-clicked advertisements in the conversion path.

As an example of tops path related to conversions for Brand X tennisshoes, a user can first be presented with an ad (“Ad 1”) for Brand X(i.e., an impression of the ad occurs), followed by the presentation ofthree consecutive ads (“Ads 2-4”) related to Brand X tennis gear, andthe presentation of an ad (“Ad 5”) for Brand X tennis shoes that theuser clicks on just before conversion. The ads can be represented, forexample, by the targeting keywords that caused their presentation (i.e.,keywords “Brand X,” “tennis” and “shoes”). In this case, the resultingconversion path including the presentation of “Ads 1-5” can berepresented as Brand X>tennis>tennis>tennis>shoes, where “Brand X” isthe keyword that caused presentation of Ad 1 and “shoes” is the keywordthat caused presentation of Ad 5. If this particular conversion path hasbeen determined to lead to the most conversions of Brand X tennis shoesover a particular period, then the path is one of the top paths. Otherconversion paths and top paths measures can be based on ads within thesame ad group, or ads in the same ad campaign.

Further, click conversion paths and impression conversion paths can begenerated, where a click conversion path represents a series of contentitem clicks without regard to impressions that occurred during theconversion cycle, and an impression conversion path represents a seriesof content item impressions without regard to the clicks that occurredprior to the last click. In any case, the conversion paths generallyrepresent a sequence of related user actions ending in the user's “last”click, such as clicking on “Ad 5,” and ultimate conversion (e.g., buyinga pair of Brand X tennis shoes).

The user interactions of a conversion path can be represented bykeywords that caused presentation of the presented and/or clickedcontent items, URLs of web pages with which the content items werepresented, and/or interaction indicators that specify whether thecontent item was presented and/or selected. Representations of the userinteractions can be presented, for example, according to a sequence inwhich the user interactions occurred. For example, the conversion pathassociated with the user's purchase of Brand X tennis shoes can beillustrated by a sequential listing of the keywords that causedpresentation of the content items.

The process 500 is a process by which one or more conversions areselected from user interaction data that specify user interactions withcontent items. In turn, a top paths (i.e., the conversion paths thatresulted in the most conversions) are then determined for each of theconversions using impressions and/or clicks that are associated with theconversion that are included in the conversion path for the conversionTop paths measures can be presented, for example, in a top paths report.

The process 500 is described below with reference to advertisingcampaigns that control distribution of advertisements in an onlineenvironment. However, the process 500 can also be used with othercontent distribution campaigns that control distribution of othercontent (e.g., video, audio, or other content).

The process 500 can be implemented, for example, by the performanceanalysis apparatus 120 of FIG. 1. In some implementations, theperformance analysis apparatus 120 is a data processing apparatus thatincludes one or more processors that are configured to perform actionsof the process 500. In other implementations, a computer readable mediumcan include instructions that, when executed by a computer, cause thecomputer to perform actions of the process 500.

Top paths measures can be determined for various types of userinteractions. For example, impression top paths measures and a click toppaths measures can be separately computed using the method 500. As usedthroughout this document, “top paths based on clicks” describes thesequence of search ad clicks (or other ad clicks) leading up toconversion. Similarly, “top paths based on impressions” describes thesequence of ad impressions leading up to conversion. In either of thetop paths measures (i.e., click- or impression-based), the clicks (orimpressions) can be measured at the keyword, ad group, and campaignlevel. In some implementations, keyword analysis of adjacent keywords inthe conversion path can be performed to remove redundant information,such as if a keyword is repeated multiples times in succession. Forexample, when keyword A is repeated in a conversion path (e.g., A>A>A>C)repetitive instances of the same keyword can be removed so theconversion path represents keyword transitions (e.g., A>C representing atransition from keyword A to keyword C).

A similar analysis can be performed for ad groups and campaigns. As aresult, two additional “transition type” top paths measures can beproduced. “Click Top paths-Transition” describes a top paths measurethat is similar to “top paths based on clicks” except that consecutive“repeat clicks” in each conversion path are collapsed to a singleinstance. Similarly, “Impression Top Paths-Transition” describes a toppaths measure that is similar to “top paths based on impressions,”except that consecutive “repeat impressions” in each conversion path arecollapsed to a single instance. For either “transition type” measure,i.e., click-based or impression-based, for example, a conversioncontaining repeated keywords, such as the repeated keyword B in theconversion path A>B>B>B>C, results in the conversion path A>B>C. Thisprocess of collapsing multiple instances into one instance can also beused for ad groups or campaigns.

In some implementations, top path measures for conversions aredetermined using only content item impressions that did not result in aclick. In other implementations, top path measures for conversions aredetermined using a total quantity of content item impressions,irrespective of whether the impressions resulted in a click by the user.In some implementations, top path measures can include the “last click”for the conversion while in other implementations, the last click can beomitted from the top path measures.

User interaction data specifying user interactions with content items isobtained (502). As described above, content items can ads that arepresented with search results, banner ads, text ads, video content, oraudio content and user interactions can include presentations of contentitems user selections (i.e., a click) of content items, and presentationof an audio and/or video content item. In some implementations, the userinteraction data includes data representing all user interactions priorto a conversion. In other implementations, the user interaction dataincludes data representing user interactions that occurred within aspecified period of time of the conversion (e.g., within three months ofthe conversion). As described in more detail below, the user interactiondata that is obtained can include a set of conversions for one or moreadvertiser, the user identifiers representing converting users for eachof the conversions, and data representing user interactions withadvertisements during the conversion cycle and by the converting users.

In some implementations, the obtained user interaction data specify userinteractions that are associated with a particular advertiser and/oruser interactions that are associated with one or more conversions. Userinteractions that are associated with a particular advertiser are userinteractions with content items provided by (or provided for) theadvertiser. User interaction data can be associated with the advertiserby being stored with an advertiser identifier (e.g., an advertiser name,campaign name, and/or another identifier) that represents the advertiserand/or by being indexed according to the advertiser identifier orindexed to a same index key (e.g., a same conversion identifier) as theadvertiser identifier.

As described above, user interactions that are specified by the userinteraction data can also be associated with one or more conversions.For example, impressions and/or selections (i.e., clicks) of contentitems (e.g., advertisements) that occurred prior to a conversion can beidentified as user interactions that are associated with the conversionand user interaction data representing the user interactions that areassociated with the conversion can be stored with a conversionidentifier that represents the conversion and/or by being indexedaccording to the conversion or indexed to a same index key as theconversion identifier. If another conversion occurs prior to anyadditional user interactions and/or within the conversion cycle for theother conversion, the impressions and/or clicks can also be associatedwith the other conversion in a similar manner.

As described above, each conversion can also be associated with contentitems that were presented and/or selected during the conversion cyclefor the conversion as well as targeting keywords that controldistribution of the content items. In some implementations, dataspecifying a targeting keyword that was matched, and therefore, caused acontent item to be presented during the conversion cycle for theconversion can be stored at a memory location associated with theconversion and/or indexed according to a conversion identifierrepresenting the conversion. User interactions and/or conversions can beassociated with one or more ad groups and/or ad campaigns based on theassociations of the user interactions and/or conversions with targetingkeywords that belong to the respective ad groups and/or ad campaigns, asdescribed above with reference to FIGS. 3A and 4A.

In some implementations, user interaction data for conversions caninclude at least two of advertisement impressions, user selections ofadvertisements, and data representing user actions by converting usersthat constitute conversions. A minimum information requirement can beestablished so that a top paths report that is generated will includemore meaningful data than it might otherwise include. For example, ifmany of the top paths include only one targeting keyword, thisinformation may not be as useful to the user as analysis of top pathsthat include more keywords. The minimum information requirement can bespecified, for example, by the advertiser.

In some implementations, each user interaction is associated with a useridentifier that represents a converting user with whom the userinteraction occurred. For example, obtaining the user interaction datain order to generate top paths reports can automatically obtaininformation for user identifiers associated with the current conversionbeing processed. To accomplish this, indexes (e.g., on user identifiers,etc.) can be established on the historical data 119 so that only userinteraction data that are pertinent to the top paths reports areaccessed.

In some implementations, the user identifier is a cookie that isretrieved from a user device with which the user interaction wasperformed, and each pair of user device and browser has a unique cookie.For example, obtaining the user identifier can occur from a cookie thatis stored on the user's device 106 by the user's browser.

In some implementations, the conversion cycle for each conversion is aperiod that includes each user interaction that occurred prior to theconversion and occurred within a specified time of the conversion. Forexample, a threshold period of time (e.g., 30 days) can established forwhich conversion paths are to be considered. Specifically, generation oftop paths measures can look back in time for user interactions that donot exceed the threshold (e.g., are not older than 30 days). In someimplementations, different thresholds can be established for differentadvertisers, ad groups and ad campaigns.

A conversion is selected from the user interaction data (504). In someimplementations, the conversion can be selected in response to receivinga request, from an advertiser, for a top paths report (i.e., a reportthat provides top paths measures) for a content distribution campaign(e.g., ad campaign). For example, an advertiser may access a userinterface similar to that described with reference to FIG. 5B, andrequest a top paths report that specifies one or more top paths measuresfor a particular ad campaign, particular ad group, or particulartargeting keyword. In turn, conversions for computing the top pathsmeasures can be selected.

In some implementations, the conversion can be selected as part of aperiodic top paths report update. For example, a top paths report can begenerated and, in turn, updated periodically to provide an advertiserwith an updated top paths report upon demand. Pre-computing top pathsreports can provide faster access to the reports in response to therequest. The description that follows describes selecting an individualconversion, but groups of conversions can also be selected forprocessing.

In some implementations, the performance analysis apparatus 120 selectsthe conversion (e.g., a conversion related to Brand-X tennis shoes). Forexample, the performance analysis apparatus 120 can access theadvertising data store 119 and identify each conversion that isassociated with a particular advertiser for which a top paths report isbeing generated or updated. The performance analysis apparatus 120 canidentify conversions that are associated with a particular advertiser,for example, by searching the user interaction data for the advertiseridentifier representing the particular advertiser and identifying theconversions by which each instance of the advertiser identifier isindexed, as described in more detail with reference to FIG. 10.

Once the conversions that are associated with the particular advertiserhave been identified, the performance analysis apparatus 120 can selectone or more of the conversions (504). The conversions can be selectedrandomly, in an order in which they are indexed in the user interactiondata, or in another predetermined order. For example, the userinteraction data can include a set of conversions that each has a uniqueconversion identifier that uniquely identifies the conversion with whichthe identifier is associated. In this example, the conversions can beselected in sequential order of the conversion identifiers. Eachconversion can also have a timestamp that specifies a time at which theconversion occurred, and the timestamp can be used to select conversionsin chronological order in which the conversions occurred.

User interactions that are associated with the selected conversion areselected from the user interaction data (506). In some implementations,the selected user interactions are user interactions with the convertinguser prior to an action that constituted the conversion. When the userinteraction data are indexed according to the conversion with which theuser interaction data are associated, as described in more detail withreference to FIG. 10, the user interactions can be selected by examining(e.g., searching) and/or filtering the user interaction data that areindexed to the selected conversion.

In some implementations, the identified user interactions areinteractions that occurred at any time prior to the conversion. In otherimplementations, the selected user interactions can be user interactionsthat occurred during the conversion cycle, but prior to the last clickfor the conversion. In still other implementations, the identifier userinteractions are user interactions that occurred within a specified timepreceding the conversion. For example, the identified user interactionsmay include the user interactions that occurred within 30 days precedingthe conversion, but exclude user interactions that occurred prior to the30 days preceding the conversion.

The specified time can be specified by the advertiser or determined, forexample, on an advertiser specific basis according to an analysis of thelag time measures for each advertiser. In some implementations, thespecified time for each advertiser can be selected as a threshold periodthat exceeds the expected time period (e.g., based on lag time measures)of conversion cycles for at least a minimum portion of the advertiser'sconversions. For example, using the lag time measures for theadvertiser, a specified time can be selected that is likely to produceconversion paths that provide accurate top paths measures for at least67% of the advertiser's conversions.

A conversion path for the conversion is generated (508). The conversionpath specifies an order in which the user interactions from the set ofcorresponding user interactions occurred leading up to conversion. Theconversion path further specifies keywords that caused presentation ofcontent items with which the user interactions occurred. As describedabove, the conversion path can be described by a path length is computedbased on presentations of content items to the user and user clicksleading up to the conversion. For example, a path length of four canrepresent a conversion path having the impression of the Brand X ad, theuser click on a Brand X ad, the presentation of the Brand-X tennisequipment ad, and the user's (“last”) click on the ad for Brand X tennisshoes.

In some implementations, the conversion path for the conversion can begenerated as a click conversion path that represents user selections(i.e., clicks) of content during the conversion cycle. The clickconversion path can be generated using a sequence of user selections(i.e., clicks) of content items that occurred during the conversioncycle for the conversion. The click conversion path can be presented toa user by providing data that causes presentation of targeting keywordsthat caused presentation of the user selected (i.e., clicked) contentitems. The content item selections can be represented by the targetingkeywords that caused presentation of the content items that wereselected. When the click conversion path is presented, the targetingkeywords can be presented according to the sequence of content itemselections. For example, the keywords that appear in a top paths reportcan be ordered by the sequence that ad impressions associated with thosekeywords were presented to the user during the conversion path that ledto conversion.

A determination is made whether more conversions remain to be processed(510). When more conversions remain to be processed, then the process500 selects another conversion (504). For example, when a top pathsreport is being generated for a particular advertiser, the performanceanalysis apparatus 120 can determine that additional conversions remainto be processed for the particular advertiser.

When no conversions remain to be processed, then data are provided thatcauses presentation of at least one conversion path for at least oneconversion (512). The information for the one or more conversion pathscan be provided, for example, for presentation in a top paths report. Asan example, the information can be sorted in descending order ofconversions so that the top paths report can display the information forconversion paths having the most conversions listed first.

Other top paths measures can also be computed and provided. For example,the user interactions associated with each conversion can be used togenerate a histogram that represents the distribution of conversionsaccording to top paths, as described in more detail with reference toFIG. 5B.

In some implementations, the process of generating top paths reportsdetermines that a same targeting keyword caused presentation of acontent item for two or more sequential content item selections. Theprocess can represent the targeting keywords according to the sequenceof content item selections. The two or more sequential content itemselections are replaced with a single instance of the same targetingkeyword. For example, a conversion path that includes the keywords“tennis—tennis—tennis” can be replaced with “tennis.”

FIG. 5B is a screen shot of an example user interface 520 for providingtop paths measures for conversions. For example, an advertiser 108(e.g., the advertiser of Brand X shoes) can use the user interface 520to display and review information for advertisement conversion measuresand reports associated with conversion top paths. In someimplementations, the information provided in the user interface 520includes conversion information (e.g., top paths measures forconversions and associated data) provided by the advertisementmanagement system 110 and/or the performance analysis apparatus 120. Forexample, some conversion information displayed in the user interface 520can be determined using the process 500 described with reference to FIG.5A.

The user interface 520 includes a header 521 and a top paths report 522.The header 521 (e.g., “MyAdvertisingAccount—Top Paths”) identifies theinformation in the user interface 520 as relating to top paths forconversions associated with an advertising account. The top paths report522 contains the advertisement conversion information, specificallydisplaying and summarizing top paths for conversions within a specifictime period based on settings established by the user (e.g., theadvertiser 108 displaying the user interface 520).

The top paths report 522 includes a page title 523 (e.g., “Top Paths”)and a date range selection control 524. The page title 523 identifiesthe top paths report 522 as containing “Top Paths” information, which isconsistent with the subject displayed in the header 521. The date rangeselection control 524 enables the user (e.g., advertiser) to specify adate range for the assisted conversion measures that are presented inthe top paths report 522. The date range specified in the date rangeselection control 524 can be used to select data with which the toppaths measures presented in the top paths report 522 are computed. Forexample, the date range can be used to select user interaction data thatare associated with conversions that occurred within the specified daterange, user interaction data that are associated with conversion paths(or cycles) where the entire conversion path occurred during thespecified date range, or user interaction data that are associated withconversion paths where any portion of the conversion path occurredduring the specified date range.

As shown, a current time period selection 525 that has been selectedusing the date range selection control 524 is the month-long period fromMar. 16 to Apr. 15, 2010. This time period can correspond to any timeduration selected by the advertiser. For example, the user can select atime period as short as a few hours up to a time period of several daysor weeks, such as a 30-day duration, and years. Some advertisers mayselect a longer date range than others because, in general, while someadvertisement conversions can occur rapidly, even within a few seconds,others can take several days, weeks, or even years. As a result, toppaths that originate, for example, from the first ad impression that ispresented the user, can span several hours or days before a conversionoccurs.

In some implementations, when the user selects the date range selectioncontrol 524, a calendar interface (not shown) can appear. For example,the user can identify dates in the current selection 525 by clicking onindividual cells on a calendar display, such as cells for Mar. 16 andApr. 15, 2010. In some implementations, the date range selection control524 can provide additional controls for specifying time ranges onspecific dates. An example range using date and time, for example, canbe noon on Mar. 16, 2010 to 5 PM on Mar. 17, 2010. In someimplementations, various controls (e.g., sliders, etc.) can be used toallow the user to specify dates and times, without having to type invalues explicitly, for example.

The top paths report 522 includes various settings for controlling thetype of top paths report that is displayed in the user interface 520 forthe current time period selection 525. A report type control 526 caninclude, for example, a drop down menu having user selectable options,active links or other user interface elements that upon selection causesubmission of a request for one or more different types of top pathsreports.

In the present example, the report type control 526 can causepresentation of a clicks-based top paths report or an impressions-basedtop paths report. For example, a user selection of the “Keyword Path(Clicks)” option 527 can cause submission of a request for a top pathsreport based on keywords and user clicks. In another example, selectionof an “Ad Group (Impressions)” option can cause submission of a requestfor a top paths report based on ad impressions seen by the user, wherethe information is summarized by ad group. In response to thesubmission, data that causes presentation of the selected report can beprovided to the user device. As shown in FIG. 5B, the current selection527 is set to display top paths report based on keywords and clicks, andtop paths report is presented in the user interface 520.

The top paths report 522 includes a conversion type selection control528 that enables the user to specify one or more types of conversionsfor which the top paths report includes related information. In someimplementations, the conversion type selection control 528 can be a dropdown menu having user selectable options. In other implementations, theconversion type selection control 528 can be a hypertext link or anotheruser interface control element (e.g., a text box, check box, or radiobutton). In the example shown, the conversion type selection control 528indicates that the top paths report includes information for “all” typesof conversions is presented in the top paths report 522.

Top paths reports can be generated for each of many different types ofconversions. In some implementations, each of the following user actionscan be identified as a different conversion type: signing up for amailing list, making a purchase, creating a user profile, making areservation, requesting presentation of a video, requesting presentationof audio, downloading one or more files, installing one or more programson the user device, or providing specified information, and a separatetop paths report can be generated for each of these different conversiontypes

Each advertiser can specify its own set of conversion types and the useractions that constitute a conversion for each conversion type. Forexample, an on-line gaming company can identify three types ofconversions. The first conversion type can be identified as aninformation conversion that is completed when a mailing address(physical or electronic) is received from a user. The second conversiontype can be identified as a download conversion that is completed when auser downloads and/or installs a trial version of an on-line game. Thethird conversion type can be identified as a purchase conversion that iscompleted when a user purchases an on-line game. In this example, theconversion type selection control 528 enables allows the user (i.e., theadvertiser) to request presentation of an top paths report thatspecifies top paths measures for purchase conversions and/or an toppaths report that specifies top paths measures for other types ofconversions (e.g., information conversions or download conversions).

In some implementations, the top paths report 522 can separatelyspecify, in a single user interface display, top paths measures for morethan one conversion type. Continuing with the example above, in responseto a request for a top paths report that includes separate top pathsmeasures for purchase conversions and download conversions, data can beprovided that cause presentation of the top paths measures for thepurchase conversions at a display location that is adjacent to thedisplay location of the top paths measures for the download conversions.Thus, a user can directly compare the top paths measures for purchaseconversions to the top paths measures for the download conversions.

An analysis type selection control 530 allows the user to specify thetype of information displayed on the screen. The current setting for theanalysis type selection control 530 is “Top Paths.” As an example of howto change the type of information displayed in the user interface 520,the user can select a different setting (e.g., “Time Lag”) from theanalysis type selection control 530. In response to the change,information related to the different setting (e.g., time lags) isdisplayed (as described above with reference to FIG. 3B), and the header521 and the page title 523 are updated to indicate the new selection(e.g., “Time Lags”).

A table portion 531 of the top paths report 522 provides detailed toppaths information within the user interface 520. For example, the tableportion 531 shown in FIG. 5B includes data values and display areasrelated to conversion top paths. The table portion 531, as depictedusing the current settings for various user controls and settings,includes rows of information arranged in three columns. A path numbercolumn 532 labels each of the top paths in the table portion with aunique path number. A path column 534 identifies the keywords (e.g.,“golf—shoes—places”) of each of the paths top paths in the table portion531. A conversion percentages column 536 identifies the overallpercentage (e.g., relative to 100% of the conversions) that each of thepaths represents, on a row-by-row basis in the table portion 531.

For example, a row 538 in the table portion 531 identifies path number 1as “golf—shoes—places,” and 16,727 conversions occurred for that toppath. The row 538 is listed first in the table portion because no othertop path (i.e., based on keyword path and clicks) resulted in moreconversions. A row 540 identifies the path“sports—golf—vacations—Arizona” as having 6,710 conversions. As aresult, the path appears as the second entry in the table portion 531,and is identified with a “2” in the path number column 532 because it isthe conversion path that led to the second highest number ofconversions. Another example row 542, or the tenth entry in the tableportion 531, identifies the path “golf—shoes—golf—shoes” as having 620conversions, which is the tenth highest number of conversions for aconversion path.

In general, the order of keywords that appear in a row (e.g., row 540)matches the sequence that ad impressions associated with those keywordsthat were viewed by the user during the conversion path that led toconversion. For example, in the 6,710 conversions associated with thepath “sports—golf—vacations—Arizona” (e.g., in row 540), users whoeventually converted were presented ads having associated keywords“sports” through “Arizona,” and the corresponding ads appeared in thatorder.

The user interface 520 includes a display type selection control 548that can be used, for example, to select one or more types of views fordisplaying information in the conversion percentages column 536, orelsewhere in the user interface. For example, the view selection 549that is currently selected is the table option, as indicated by aspreadsheet icon. As a result, the percentages displayed in theconversion percentages column 536, appear in tabular format. Otheroptions selectable from the display type selection control 548 can causethe data in the table portion to be displayed a histogram or in otherways.

The user interface 520 includes other controls and areas. For example,an address bar 550 can identify the address (e.g., URL) that isassociated with (e.g., identifies a network location for) the data thatcauses presentation of the current display of the user's browser, inthis case top paths information for an advertising account. An exportcontrol 552 can provide the user with options for exporting informationfrom the user interface 520, primarily the information included in thetable 531 and the identification of any user settings used to generatethe data. The export control 552 can also be used to export userinteraction data and/or data that may have been computed as describedherein, but has not been presented. A scroll bar 554 is provided in someimplementations to allow the user to scroll to other parts of the userinterface 520, such as if the more top paths appear in the table 531 butare not currently within the viewport of the top paths report 522.

The top paths report 522 includes a summary 556 (e.g., “There were37,555 conversions via 84 campaign paths (clicks)”) which can be used todirect the user's attention to specific information on the userinterface 520, or to provide a summary of the information presented. Forexample, the summary 556 in this case provides a total of theconversions corresponding to the top paths and identifies that theconversions are keyword- and click-based.

Assist User Interactions

FIG. 6A is an example process 600 for tracking assist user interactionsin association with conversions and computing assist user interactionmeasures for conversions. An assist user interaction for a conversion isa user interaction with a converting user (i.e., a user that performs anaction that constitutes the conversion) during a conversion cycle forthe conversion and prior to the last click for the conversion. Forexample, an assist user interaction can be any user interaction by theconverting user that occurs prior to user selection (or presentation) ofthe last clicked advertisement for the conversion. In someimplementations, assist user interactions include one or more assistimpressions and/or one or more assist clicks. An assist click for aconversion is a selection of a content item by the converting userduring the conversion cycle for the conversion and prior to the lastclick for the conversion.

An assist impression for a conversion is a presentation of a contentitem (e.g., advertisement, video, audio, or widget, or another contentitem) to the converting user prior to the conversion. In someimplementations, assist impressions for a conversion can include onlyimpressions other than the impression of the last clicked advertisementfor the conversion. For example, if a converting user is presented asingle advertisement, selects the advertisement, and converts, thenthere will be no assist impressions for that conversion because the onlyadvertisement that was presented was the last clicked advertisement.When a conversion occurs without any prior clicks, such that there isnot a last clicked advertisement, each impression during the conversioncycle for the conversion can be an assist impression for the conversion.

The process 600 is a process by which one or more conversions areselected from user interaction data that specifies user interactionswith content items. In turn, user identifiers that are associated witheach of the conversions are identified and quantities of assist userinteractions is determined for each of the user interactions, forexample, using the user identifiers to identify assist user interactionsfor each of the conversions. A measure of user interactions is thencomputed using the quantities of assist user interactions for theconversions.

The process 600 is described below with reference to advertisingcampaigns that control distribution of advertisements in an onlineenvironment. However, the process 600 can also be used with othercontent distribution campaigns that control distribution of othercontent (e.g., video, audio, or other content).

The process 600 can be performed, for example, by the performanceanalysis apparatus 120 shown in FIG. 1. In some implementations, theperformance analysis subsystem 120 is a data processing apparatus thatincludes one or more processors that are configured to perform actionsof the process 600. In other implementations, a computer readable mediumcan include instructions that, when executed by a computer, cause thecomputer to perform actions of the process 600. The stages shown in FIG.6 are illustrative of a set of stages that can be used to perform theprocess 600. Additional or fewer stages may be used, or stages may notbe used at all.

User interaction data specifying user interactions with content items isobtained (602). The content items can be, for example, advertisements(ads) displayed with search results provided by a search system inresponse to a search query that was submitted by a user. As anotherexample, the content items can include banner ads displayed with a webpage provided by a publisher. As yet another example, the content itemscan be or include video or audio content. Forms of user interactions caninclude presentation of a content item (i.e., an impression) to a user,selection (i.e., a click) of a content item by a user, presentation ofall or a portion of an audio and/or video content item to the user. Insome implementations, the user interaction data are collected over aspecified time period. For example, user interaction data can becollected for all user interactions that have occurred over the last twomonths. As another example, user interaction data are collected for alluser interactions associated with conversions that have occurred overthe past 15 days. In some implementations, the user interaction datathat are obtained are associated with multiple users. In otherimplementations, the user interaction data that are obtained isassociated with a single user.

In some implementations, the obtained user interaction data specify userinteractions that are associated with a particular advertiser. Userinteractions that are associated with a particular advertiser are userinteractions with content items provided by the advertiser. For example,user interaction data can be obtained that represent user interactionswith all content items that are provided by and/or provided for aparticular travel services company (e.g., Travel Planning Corp.(“TPC”)). User interaction data that represent user interactions can beassociated with the advertiser by being stored with an advertiseridentifier that represents the advertiser and/or by being indexedaccording to the advertiser identifier or to a same index key (e.g., asame conversion identifier) as the advertiser identifier.

In some implementations, user interactions are associated with one ormore conversions. For example, a user that performed one or more userinteractions (i.e., was presented one or more advertisements and/orselected one or more of the presented advertisements) can subsequentlysign up for a mailing list for TPC. In this example, when TPC. specifiesthat signing up for a mailing list is a conversion, then the impressionsand selections of content items (e.g., advertisements) for TPC thatoccurred with respect to the user prior to the conversion by the usercan be identified as user interactions that are associated with theconversion.

Continuing with this example, the user interaction data can specify thatprior to the conversion the user was presented a first ad and a secondad for TPC and that the user selected the second ad, but did not selectthe first ad. In this example, the user interaction data can specifythat the two interactions (i.e., one impression and one selection) areassociated with the conversion, for example, by storing datarepresenting the user interactions with references to the conversionand/or indexing the user interactions according to a conversionidentifier for the conversion. In this example, since the user selectedthe second ad, the presentation of the second ad is not counted as anassist impression for the conversion. In some alternativeimplementations, the user interaction data can specify that threeinteractions (i.e., two impressions and one selection) occurred. In suchimplementations, the presentation of the second ad can be counted as aninteraction (i.e., an impression) separately from the selection of thesection ad (i.e., a selection).

In some implementations, content items are associated with one or moretargeting keywords that control distribution of the content items, asdescribed above with reference to FIG. 1. Each user interaction with acontent item can be associated with one or more targeting keywords thatare associated with the content item. In some implementations, each userinteraction with a content item is associated with a targeting keywordthat was matched, and therefore, caused the content item to bepresented. For example, if the keyword “clothing” is used to select anad for presentation to a user and the user subsequently selects (i.e.,clicks) the identified ad, the keyword “clothing” can be associated withthe user selection. If the user selection of the ad is the last userselection of an ad provided by the advertiser prior to a conversion, thekeyword “clothing” can be associated with a last click for theconversion.

As another example, if an advertisement for a hotel chain is the lastadvertisement selected by a user prior to the user booking a room withthe hotel chain, the advertisement can be identified as a last clickedadvertisement for the conversion. Additionally, the conversion can beassociated with the advertisement and with a keyword (e.g., “hotels inAtlanta”) that caused the advertisement to be selected for presentation.As yet another example, if an advertisement is selected for presentationbased on the targeting keyword “toy cars,” being matched, userselections of the advertisement will be associated with the keyword “toycars.” In this example, each presentation of the advertisement to theuser where the user did not select the advertisement can also beassociated with the keyword “toy cars” as an impression.

In some implementations, the user interactions that are associated witheach conversion include one or more presentations of content items to auser (i.e., impressions) and/or one or more user selections of thecontent items by the user prior to the conversion occurring. Forexample, a user can be presented a first ad for a pizza chain and laterbe presented a second ad for a pizza chain without selecting either ofthe two ads. The user can then type in a URL for the website of thepizza chain and order a pizza online. The impressions (i.e., thepresentations of the two ads by the user) can be associated with theconversion (i.e., the purchasing of the pizza by the user). In someimplementations, the two impressions are identified as assistimpressions for the conversion, as described in more detail below. Inthis example, user interactions that are associated with the conversioninclude the two impressions and no selections (i.e., clicks). Thus, twouser interactions are associated with the conversion.

As another example, the user can be presented the first ad and thesecond ad. Assuming that the user does not select the first ad, butselects the second ad, selection of the second ad can cause the user tobe redirected, for example, to the website for the pizza chain. When theuser orders a pizza on the website, a conversion is associated with theuser. In this example, the selection of the second ad can be identifiedas a last click for the conversion and the presentation of the first adcan be identified as an assist impression for the conversion. In someimplementations, a keyword that is associated with the second ad andcaused presentation of the second ad is associated with data identifyingthe keyword as the keyword that caused presentation of a last clickedcontent item for the conversion. A keyword that is associated with thefirst ad and caused presentation of the first ad is associated with anassist impression for the conversion.

A conversion is selected from the user interaction data (604). Forexample, user interaction data associated with TPC can include userinteraction information associated with multiple conversions. One ormore of the conversions can be selected from among the multipleconversions that are included in the user interaction data. Conversionscan be selected randomly, in an order in which they are indexed in theuser interaction data, or in a predetermined order. For example, theuser interaction data can include a set of conversions that each has aunique conversion identifier that uniquely identifies the conversionwith which the identifier is associated. In this example, theconversions can be selected in sequential order of conversionidentifiers. Each conversion can also have a timestamp that specifies atime at which the conversion occurred. Thus, conversions can be selectedin chronological order in which the conversions occurred.

As another example, conversions can be arranged according to keywordsthat are associated with last clicks for the conversions. For example,all conversions for which the last clicked ad was selected forpresentation using the targeting keyword “car rental” can be groupedtogether. In this example, each conversion that is associated with alast clicked keyword (i.e., a keyword that caused presentation of a lastclicked content item) of “car rental” can be selected as a group.

A quantity of assist user interactions for the conversion is determined(606). As described above, the assist user interactions can include oneor more assist impressions and/or one or more assist clicks. In someimplementations, each content item presentation during the conversioncycle for the conversion qualifies as an assist impression for theconversion irrespective of whether the user selected the presentedcontent item. For example, if a first advertisement is presented to theuser and selected by the user, the user interactions with the contentitem would include an assist impression.

In other implementations, impressions qualify as assist impressions whena content item is presented to a user but not selected by the user. Inthese implementations, if an ad is presented to the user and the userdoes not click on the ad, the impression can be identified as an assistimpression. However, if an ad is presented to the user and the userclicks on the presented ad, the user interaction with the ad does notqualify as an assist impression, but may qualify as an assist click, asdescribed below.

Assist user interactions can also include assist clicks. An assist clickis a user selection of a content item that occurs during a conversioncycle for the conversion, and prior to the last click for theconversion. As described above, the last click for a conversion is alast user selection of a content item preceding the conversion by theuser. For example, three different ads for a shoe brand are presented toa user, the user selects each of the three ads, and purchases a pair ofshoes after selecting the last of the three presented ads. The lastclick for this conversion is the selection of the last of the threepresented ads, while the selections of each of the first two ads areassist clicks. Note that the conversion cycle for this conversion (i.e.,the purchase of the shoes) includes two assist clicks, one last clickand one conversion.

A single content item can be associated with multiple assist userinteractions for a single conversion cycle. For example, a first ad ispresented to a user twice and the user selects the ad the first time itis presented, but does not select the ad the second time it ispresented. In this example, a conversion does not occur after the firstor second presentation of the ad. A second ad associated with the sameadvertiser as the second ad is then presented. The user selects thesecond ad and makes a purchase from the advertiser (i.e., a conversionoccurs). The first ad is associated with both an assist impression andan assist click for the conversion cycle. The assist impression and theassist click can be associated with the first ad by being stored with anad identifier that represents the ad and/or by being indexed accordingto the ad identifier or to a same index key (e.g., a same conversionidentifier) as the ad identifier.

In some implementations, keywords that cause presentation of the contentitems with which the user interactions occur are considered associatedwith the user interactions. For example, a keyword that causespresentation of a content item associated with an assist click can beidentified as an assist click keyword for a conversion and a keywordthat causes presentation of a content item associated with an assistimpression can be identified as an assist impression keyword for theconversion. Continuing with the example above, the first ad for the shoebrand can be selected for presentation based on the targeting keyword“shoes,” being matched by a search query, while the second ad for theshoe brand can be selected for presentation based on the targetingkeyword “running” being matched by another search query, and the thirdad for the shoe brand can be selected for presentation based on thetargeting keyword “hiking” being matched by a third search query. Inthis example, the keywords “shoes” and “running” are each associatedwith an assist click because each of these targeting keywords causedpresentation of an ad that was selected by the user during theconversion cycle for the conversion prior to the last click for theconversion cycle, while the keyword “hiking” is associated with a lastclick because the keyword “hiking” caused presentation of the ad thatwas last selected prior to the conversion.

A single keyword can be associated with multiple assist user actionsduring a single conversion cycle. For example, if the three adsdescribed above are each selected for presentation based on the keyword“running shoes” being matched, the keyword “running shoes” can beassociated with two assist clicks and one last click. A keyword can beassociated an assist user interaction by being stored with a userinteraction identifier that represents the user interaction and/or bybeing indexed according to the user interaction identifier or to a sameindex key (e.g., a same conversion identifier) as the user interactionidentifier.

In some implementations, data representing user interactions that areassociated with a conversion are indexed according to the conversionwith which the user interactions are associated. As described in moredetail below with reference to FIG. 10, each conversion can be stored ata separate row (or column) of a database and data that are associatedwith the conversion can be stored at separate locations of the same row(or column) as the conversion. In these implementations, the userinteractions for each conversion can be identified, extracted, and/orobtained from locations (e.g., columns) of the row in which theconversion is stored.

In other implementations, assist user interactions are identified foreach conversion using the user identifier that is associated with theconversion. For example, a user performs an act that constitutes aconversion (e.g., purchasing a product, making a reservation, signing upfor a mailing list, etc.) and a user identifier for the converting useris associated with data representing the conversion. The user identifierfor the converting user can then be used to search the user interactiondata to identify impressions and clicks that are associated with theuser identifier for the converting user. For example, if the conversionis a purchase of Brand-X hand sanitizer, the user identifier that isassociated with the purchase can be used to identify four impressionsthat are associated with the converting user and occurred prior to theconversion. The user identifier can also be used to identify clicks, bythe converting user, of one or more ads that were presented during theconversion cycle for the conversion.

A determination is made as to whether there are more conversions toprocess (608). If there are additional conversions to process, theprocess 600 returns to stage 604 of selecting a conversion from the userinteraction data. If it is determined that there are no remainingconversions to process, the process 600 proceeds to stage 610.

A measure of assist user interactions for the advertiser is computedusing the quantity of assist user interactions for the conversions(610). Measures of assist user interactions (“assist interactionmeasures”) can be computed on a per-advertiser, per content item,per-advertising campaign, per-ad group, and per-keyword basis. Themeasures of assist user interactions can also be computed for otherattributes of an advertising campaign (e.g., a per-bid basis or aper-budget basis).

Measures of assist user interactions can include a total quantity ofassist clicks and/or a total quantity of assist impressions. Themeasures of assist user interactions can also include an averagequantity of assist clicks per conversion and/or an average quantity ofassist impressions per conversion. Additionally, other statisticalmeasures (e.g., standard deviation, mode, median) and distributions ofassist clicks and/or assist conversions can be generated. The discussionbelow describes several example measures of user interactions that canbe computed, but other measures can be computed.

In some implementations, the measure of assist user interactions can beassociated with the advertiser, with a content item that is distributedfor the advertiser, and/or with a keyword that controls distribution ofcontent items for the advertiser. For example, a measure of assist userinteractions can be computed for all content that is distributed forTPC. As another example, a measure of assist user interactions can becomputed for an ad that is distributed for TPC. As yet another example,a measure of assist user interactions can be computed for a keyword thatis used to control distribution of one or more ads for TPC

Computing a measure of assist user interactions for the advertiser caninclude determining a total quantity of assist user interactions thatare associated with conversions for the advertiser. For example, thequantity of assist user interactions determined at stage 606 can besummed for all conversions that are associated with the advertiser. Forexample, assume that three conversions occurred for an advertiser duringa specified time period (e.g., 1 month). Assume further that theconversion cycle for the first conversion includes 3 assist userinteractions, the conversion cycle for the second conversion includes 2assist user interactions, and the conversion cycle for the thirdconversion includes 5 assist user interactions. In this example, thetotal quantity of assist user interactions for the advertiser during thespecified time period is 10.

In some implementations, computing a measure of assist user interactionsfor the advertiser includes computing a total quantity of assist clicksassociated with the advertiser. For example, the quantity of assistclicks associated with each conversion for the advertiser can be summedto identify a total quantity of assist clicks for the advertiser.

Computing a measure of assist user interactions (“assist interactionmeasures”) for the advertiser can include computing an average quantity(or another statistical measure) of assist clicks for each conversion.For example, if the total quantity of assist clicks associated with theadvertiser (or ad group or keyword) is 36 and the total quantity ofconversions associated with the advertiser (or ad group or keyword) is12, the average quantity of assist clicks per conversion can becalculated by dividing 36 by 12 to give an average of 3 assist clicksper conversion.

In some implementations, computing a measure of assist user interactionsfor the advertiser includes computing a total quantity of assistimpressions associated with the advertiser. For example, the quantity ofassist impressions associated with each conversion is summed to identifya total quantity of assist impressions for the advertiser. In someimplementations, computing a measure of assist user interactions for theadvertiser includes computing an average quantity of assist impressionsfor each conversion. For example, if the total quantity of assistimpressions associated with the advertiser is 26 and the total quantityof conversions associated with the advertiser is 8, the average quantityof assist impressions for each conversion is 3.25 assist impressions perconversion.

In some implementations, computing a measure of assist user interactionsfor the advertiser includes computing a ratio of assist clicks, assistimpressions, or assist user interactions to last clicks for theadvertiser. For example, if the total quantity of last clicks for theadvertiser is 342 and the total quantity of assist clicks for theadvertiser is 71, the ratio of assist clicks to last clicks for theadvertiser is 0.21.

Computing a measure of assist user interactions for the advertiser caninclude computing a total quantity of assist clicks for one or moretargeting keywords associated with the advertiser. Computing a totalquantity of assist clicks for a targeting keyword can includeidentifying the quantity of user selections of content items withinconversion cycles occurring prior to a last click where the contentitems are presented in response to the keyword being matched. Forexample, a keyword of “fashion” can cause presentation of multipleadvertisements. The total quantity of times that the advertisements arepresented in association with a match for the keyword “fashion” and theadvertisement is subsequently selected by a user prior to a lastselection (i.e., a last click) for a conversion cycle can be identified.

As another example, the keyword “fashion” is associated with twoadvertisements. The first advertisement is also associated with akeyword of “dresses.” The first advertisement is selected by a user,prior to a last click, in a conversion cycle 5 times in association withkeyword “fashion” and 3 times in association with the keyword “dresses.”The second advertisement is selected by a user, prior to a last click,in a conversion cycle 4 times in association with the keyword fashion.The total quantity of assist clicks for the keyword fashion is 9, whilethe total quantity of assist clicks for the first advertisement is 8 andthe total quantity of assist clicks for the second advertisement is 4.

Computing a measure of assist user interactions for the advertiser caninclude computing a total quantity of assist impressions for one or morekeywords associated with the advertiser. Computing a total quantity ofassist impressions for a keyword can include identifying the quantity oftimes a keyword causes content items to be presented prior to aconversion occurring, where the content items are not selected.Computing a measure of assist user interactions for the advertiser caninclude computing a ratio of assist clicks or assist impressions to lastclicks for one or more keywords, where the quantity of last clicks for akeyword can be a quantity of last clicks for content items that werepresented in response to the targeting keyword being matched. Forexample, a keyword can be identified as being associated with 12 assistclicks and 3 last clicks. The ratio of assist clicks to last clicks forthe keyword is 4.

Computing a measure of assist user interactions for the advertiser caninclude computing a total quantity of assist clicks and/or a totalquantity of assist impressions for one or more advertising campaigns oradvertisement groupings associated with the advertiser. An advertisingcampaign for a particular advertiser can include a quantity ofadvertisements directed toward the same or similar products. Forexample, a shoe company can have an advertising campaign that includesseven different ads for a particular running shoe. The quantity ofassist clicks and assist impressions for the advertising campaign can becalculated by summing the quantity of assist clicks and assistimpressions for each ad included in the ad campaign. In someimplementations, computing a measure of assist user interactions for theadvertiser includes computing a ratio of assist clicks or assistimpressions to last clicks for one or more advertising campaigns.

Computing a measure of assist user interactions for the advertiser caninclude computing a total quantity of assist clicks and/or a totalquantity of assist impressions for a content item that is distributedfor the advertiser. For example, an advertisement is identified asappearing in conversion cycles for three conversions. In the firstconversion cycle, the advertisement is viewed but not clicked once, andclicked prior to the last click once. In the second conversion cycle,the advertisement is clicked twice prior to the last click. In the thirdconversion cycle, the advertisement is viewed but not clicked twice andclicked once as the last click prior to the conversion. In this example,the total quantity of assist clicks for the advertisement is three andthe total quantity of assist impressions for the advertisement is three.In some implementations, the total quantity of assist clicks and/or thetotal quantity of assist impressions is computed for all content itemsassociated with the advertiser. In other implementations, the totalquantity of assist clicks and/or the total quantity of assistimpressions is computed for less than all of the content itemsassociated with the advertiser.

Computing a measure of assist user interactions for the advertiser caninclude computing a ratio of assist clicks to last clicks for one ormore content items where the quantity of last clicks for a content itemcan be a number of times the content item was the last click in aconversion cycle prior to a conversion. For example, an advertisementcan be identified as being associated with 7 assist clicks and 3 lastclicks. The ratio of assist clicks to last clicks for the advertisementis 2.33. In some implementations, computing a measure of assist userinteractions for the advertiser includes computing a ratio of assistimpressions to last clicks for one or more advertisements. For example,an advertisement can be identified as being associated with 15 assistimpressions and 3 last clicks. The ratio of assist impressions to lastclicks for the advertisement is 5.

In some implementations, data representing one or more computed measuresof assist user interactions is presented to a user. For example, thetotal quantity of assist clicks for a quantity of target keywordsassociated with the advertiser can be displayed to the user. As anotherexample, a webpage showing the ratio of assist user interactions to lastclicks for a quantity of advertisements is displayed to the user. Thedata can be presented to the user, for example, in the form of a graph,chart, table, or spread sheet. The data can be presented as part of awebpage, or exported as a file (e.g., a text file, word processing file,spread sheet, or database file).

FIG. 6B shows an assist user interaction report user interface 620 thatdepicts a browser which includes a user interface for providing assistclick and assist impression associated measures. The browser can beused, for example, to display web pages and other resources that make upwebsites, (such as the websites 104 of FIG. 1). The assist userinteraction report user interface 620 includes a header 621 forindicating information that is displayed in the assist user interactionreport user interface 620. In this example, the header 621 indicatesthat information associated with a website titled “MyAdvertisingAccount”is being presented and that the presented information more specificallyrelates to assist interactions.

The assist user interaction report user interface 620 includes an assistuser interaction report 622 for presenting information relating toconversion path performance measures and several selection controls toallow a user to specify parameters for the conversion path performancemeasures. The assist user interaction report 622 can be, for example, awebpage received from a web server through the Internet. As anotherexample, referring to FIG. 1, the assist user interaction report 622 canbe a resource 105 of a website 104 received through the network 102 byone of the user devices 106.

The assist user interaction report user interface 620 includes anaddress bar 650 that displays a URL for the assist user interactionreport 622. In some implementations, the user can access the assist userinteraction report 622 by entering the URL into the address bar 650.

The assist user interaction report user interface 620 further includes ascroll bar 654 to allow the user to scroll through content presented onthe assist user interaction report 622. The assist user interactionreport 622 includes a page title 623 indicating that the informationincluded on the assist user interaction report 622 relates to assistinteractions. The assist user interaction report 622 further includes adate range selection control 624 to allow the user to specify a daterange for conversion path performance measures to be presented on theassist user interaction report 622. In some implementations, the daterange specified in the date range selection control 624 limits theinformation presented on the assist user interaction report 622 toinformation relating to conversions that occurred within the specifieddate range. In some implementations, the specified date range limits theinformation presented on the assist user interaction report 622 toinformation relating to conversion cycles where the entire conversioncycle occurred during the specified date range. In some implementations,the specified date range limits the information presented on the assistuser interaction report 622 to information relating to conversion cycleswhere any portion of the conversion cycle occurred during the specifieddate range. For example, information relating to a conversion cyclewhere a first assist impression occurred during the specified date rangecan be included in the information presented on the assist userinteraction report 622.

The assist user interaction report 622 includes links 625 and 626 toallow the user to select a type of assist user interaction informationto present. In the example shown, selecting the link 625 causesinformation relating to assist clicks to be presented. Selecting thelink 626 causes information relating to assist impressions to bepresented. In the example shown, the link 625 is bold to indicate thatthe link 625 is currently selected and that information presented on theassist user interaction report 622 is related to assist clicks. In someimplementations, the assist user interaction report 622 includes a linkthat allows the user to view information relating to both assist clicksand assist impressions.

The assist user interaction report 622 includes a conversion typeselection control 628 to allow the user to indicate one or more types ofconversions to for which to present related information. In the exampleshown, the conversion type selection control 628 indicates thatinformation relating to all types of conversions is presented on theassist user interaction report 622. In some implementations, conversionscan be divided into different types. User actions that can be identifiedas different conversion types can include, signing up for a mailinglist, making a purchase, creating a user profile, making a reservation,watching a video, listening to an audio file, downloading one or morefiles, installing one or more programs, or providing information. Forexample, a computer software company can identify three types ofconversions. The first conversion type can be identified as a userproviding a mailing address. The second conversion type can beidentified as a user downloading and installing a trial version of aprogram. The third conversion type can be identified as a userpurchasing a full version of a program. The conversion type selectioncontrol 628 allows the user to view assist interaction measures forrevenue generating conversions (e.g., purchases) from other types ofconversions (e.g., signing up for a mailing list or downloading trialsoftware). In some implementations, the conversion type selectioncontrol 628 can be a drop down menu having user selectable options.

The assist user interaction report 622 includes an analysis typeselection control 630 to allow the user to select from among multipletypes of conversion path performance measures. In the example shown, theanalysis type selection control 630 indicates that information currentlypresented on the assist user interaction report 622 relates to assistinteractions. Other types of conversion path performance measures caninclude time lag measures, path length measures, assist interactionmeasures, last click measures, first click measures, and assistedconversion measures. In some implementations, the analysis typeselection control 630 can be a drop down menu having user selectableoptions.

As mentioned above, the assist user interaction report 622 includesconversion path performance measures as specified by the user using theselection controls and links of the assist user interaction report 622.In the example shown, the assist user interaction report 622 includesinformation relating to assist clicks associated with all conversiontypes for a date range of Mar. 16, 2010 to Apr. 15, 2010. Theinformation presented on the assist user interaction report 622 includesa list of keywords 632. The list of keywords 632 can be, for example,targeting keywords that control distribution of content items associatedwith an advertiser. As another example, the keywords 632 can betargeting keywords that control distribution of a single content item.As yet another example, the keywords 632 can be targeting keywords thatcontrol distribution of advertisements included in a specifiedadvertisement campaign or advertisement group. In the example shown, theassist user interaction report 622 presents assist click statisticsrelating to each of the keywords 632. In some implementations, in placeof or in addition to the list of keywords 632, the assist userinteraction report 622 can include a list of advertisements associatedwith an advertiser or ad campaign and show assist click statistics foreach listed advertisements. In some implementations, the assist userinteraction report 622 can include a list of ad campaigns or ad groupsand show assist click statistics for each listed ad campaign or adgroup. In some implementations, the assist user interaction report 622can include a selection control to allow the user to select a categoryof information for which to present statistics (e.g., keywords,advertisements, ad campaigns, or ad groups).

The assist user interaction report 622 further includes a list of assistclick totals 634 indicating a total quantity of assist clicks associatedwith each keyword 632 for the date range specified in the date rangeselection control 624. The assist click totals indicate that a keyword“vacation” is associated with 890 assist clicks. As described above,this indicates that the keyword “vacation” caused presentation of acontent item 890 times where the content item was selected by a userprior to a last click for a conversion path. In some implementations,selecting the link 626 causes the list of assist click totals 634 to bereplaced by a list of assist impression totals. In some implementations,selecting a link for assist user interactions causes the list of assistclick totals 634 to be replaced by a list of assist user interactiontotals.

The assist user interaction report 622 includes a bar graph 636 thatindicates assist click percentages for each of the keywords in relationto the total quantity of assist clicks for the specified date range. Thebar graph 636 visually indicates the percentage of assist clicksattributed to each keyword in relation to the total quantity of assistclicks to allow the user to readily visually inspect the totals andidentify the relative contributions of each keyword from an assist clickperspective. For example, the user can readily identify that the keyword“vacation” is associated with more assist clicks than the keyword“flights” since the bar at row 638 is larger than the bar at row 640. Asanother example, the user can readily identify that the keywords“vacation,” “flights,” and “travel” are associated with significantlymore assist clicks than the remaining keywords 632 since the barsassociated with the keywords of “vacation,” “flights,” and “travel” aresignificantly longer than the bars associated with the remainingkeywords 632. This information can assist the user identifying the mostsuccessful keywords 632 for use in on-going and future advertisingcampaigns. In some implementations, selecting the link 626 causes thebar graph 636 to present assist impression percentage information. Insome implementations, selecting a link for assist user interactionscauses bar graph 636 to present percentage information for all assistuser interactions.

In some implementations, additional assist user interaction measures(“assist interaction measures”) can be presented on the assist userinteraction report 622. For example, the assist user interaction report622 includes text 642 indicating that the total quantity of assistclicks leading to conversions for the relevant conversion paths is 2,287assist clicks. As another example, a percentage of assist clicks to lastclicks for each keyword can be presented. As another example, apercentage of assist impressions to last clicks for each keyword can bepresented. As yet another example, a percentage of assist userinteractions to last clicks for each keyword can be presented. In someimplementations, the assist user interaction measures can be presentedin additional or alternative formats. For example, a pie graph similarto that shown in FIG. 3C can be presented showing the percentage oftotal assist clicks for each keyword. As another example, a line graphshowing the total quantity of assist clicks or assist impressions forunique time periods (e.g., days or hours) over the course of thespecified date range can be included on the assist user interactionreport 622. As another example, the assist user interaction report 622can include a table listing assist user interaction measures.

The assist user interaction report 622 includes a display type selectioncontrol 648 to allow the user to select one or more formats forpresenting conversion path performance measures on the assist userinteraction report 622. In the example shown, a selection control 649 isselected. The selection control 649 is used to select a bar graph formatfor presentation of some or all of the conversion path performancemeasures presented on the assist user interaction report 622. In theexample shown, the display type selection control 648 additionallyincludes options for table, pie chart, and relative comparison chartformats. Selecting the relative comparison chart format can, forexample, cause a bar graph comparing the total quantity of assist clicksfor each keyword 632 to the average quantity of assist clicks for allkeywords associated with an advertisement management system to bedisplayed.

The assist user interaction report 622 includes an export control 652that allows the user to export some or all of the assist userinteraction measures displayed on the assist user interaction report622. In some implementations, the export control 652 allows the user tospecify an export format. For example, the user can export the measuresas a spread sheet file. As another example, the user can export themeasures in a word processing document.

Assisted Conversions

FIG. 7A is an example process 700 for computing assisted conversionmeasures. An assisted conversion is a conversion for which one or moreassist user interactions occur during a conversion cycle for theconversion. For example, a conversion that is associated with an assistimpression and/or an assist click, as described above, is an assistedconversion. In some implementations, a conversion having a conversioncycle that includes a last click, but no assist impressions or assistclicks is not an assisted conversion. A click assisted conversion is aconversion for which one or more assist user clicks occur during aconversion cycle for the conversion. An impression assisted conversionis a conversion for which one or more assist impressions occur during aconversion cycle for the conversion.

In some implementations, a keyword is identified as assisting aconversion (i.e., being an assist keyword) when the keyword causes oneor more content items (e.g., ads) to be presented during a conversioncycle for the conversion. Similarly, ad campaigns and content itemgroups (e.g., ad groups) to which the targeting keyword belongs can alsobe identified as assisting the conversion. Content items that arepresented during the conversion cycle for the conversion can also beidentified as assisting a conversion in a similar manner.

For example, if a keyword and/or an ad that are included in an adcampaign are associated with an assist click for a conversion, the adcampaign is identified as assisting the conversion and the keyword isreferred to as an “assist click keyword” for the conversion. Continuingwith this example, if another keyword and/or another ad included in thead campaign are associated with an assist impression for a conversion,the other keyword is referred to as an “assist impression keyword” andthe ad campaign is associated with an impression assisted conversion andone click assisted conversion. An assisted conversion can be associatedwith an ad campaign by being stored at a memory location associated withthe ad campaign, stored in association with an ad campaign identifierthat represents the ad campaign, and/or by being indexed according tothe ad campaign identifier or to a same index key (e.g., a sameconversion identifier) as the ad campaign identifier.

The process 700 is a process by which one or more conversions areselected from user interaction data that specifies user interactionswith content items. In turn, content items presented and/or selectedduring a conversion cycle for a selected conversion are identified andtargeting keywords that caused presentation of the identified contentitems are selected. A quantity of conversions for which the targetingkeywords caused content items to be presented is determined, and anassisted conversion measure is computed using the quantities ofconversions for which the targeting keywords caused content items to bepresented.

The process 700 is described below with reference to advertisingcampaigns that control distribution of advertisements in an onlineenvironment. However, the process 700 can also be used with othercontent distribution campaigns that control distribution of othercontent (e.g., video, audio, games, or other content).

The process 700 can be performed, for example, by the performanceanalysis apparatus 120 shown in FIG. 1. In some implementations, theperformance analysis apparatus 120 is a data processing apparatus thatincludes one or more processors that are configured to perform actionsof the process 700. In other implementations, a computer readable mediumcan include instructions that, when executed by a computer, cause thecomputer to perform actions of the process 700. The stages shown in FIG.7A are illustrative of a set of stages that can be used to perform theprocess 700. Additional or fewer stages may be used, or stages may notbe used at all.

User interaction data specifying user interactions with content items isobtained (702). The content items can be, for example, ads presentedwith search results, banner ads, text ads, video content, or audiocontent. Forms of user interactions can include presentation of acontent item (i.e., an impression) to a user, selection (i.e., a click)of a content item by a user, and presentation of all or a portion of anaudio and/or video content item to the user. In some implementations,the user interaction data are collected over a specified time period.For example, user interaction data can be collected for all userinteractions that have occurred over the last two months (or otherspecified time periods).

In some implementations, the obtained user interaction data specify userinteractions that are associated with a particular advertiser. Userinteractions that are associated with a particular advertiser are userinteractions with content items provided by the advertiser. Userinteraction data that represent user interactions are associated withthe advertiser by being stored with an advertiser identifier (e.g., anadvertiser name, campaign name, and/or another identifier) thatrepresents the advertiser and/or by being indexed according to theadvertiser identifier or to a same index key (e.g., a same conversionidentifier) as the advertiser identifier.

User interactions that are specified by the user interaction data can beassociated with one or more conversions. For example, a user with whichone or more user interactions occurred (i.e., a user that was presentedone or more advertisements and/or selected one or more of the presentedadvertisements) can subsequently purchase goods from a retailer (i.e.,complete a conversion). In this example, the impressions and/orselections of content items (e.g., advertisements) that occurred priorto the conversion can be identified as user interactions that areassociated with the conversion, and are respectively referred to asassist impressions and assist clicks, as described above with referenceto FIG. 6A. If another conversion occurs prior to any additional userinteractions and/or within the conversion cycle for the otherconversion, the assist impressions and assist clicks described above canalso be associated with the other conversion.

For example, a user can be presented a first ad for a pizza chain andlater be presented a second ad for a pizza chain without selectingeither of the two ads. The user can then submit the URL for a web pageof the pizza chain and order a pizza online (i.e., perform an act thatconstitutes a conversion). In this example, the impressions (i.e., thepresentations of the two ads by the user) can be associated with theconversion (i.e., the purchasing of the pizza by the user) as assistimpressions for the conversion, as described above. If the user againsubmits the URL of the web page and orders another pizza (or performsanother act that constitutes a conversion) the impressions can also beassociated with a second conversion, assuming that the impressionsoccurred within the conversion cycle for the second order.

As described above, each conversion can also be associated with contentitems that were presented and/or selected during the conversion cyclefor the conversion as well as targeting keywords that controldistribution of the content items. In some implementations, dataspecifying a targeting keyword that was matched, and therefore, caused acontent item to be presented during the conversion cycle for theconversion can be stored at a memory location associated with theconversion and/or indexed according to a conversion identifierrepresenting the conversion.

For example, if the targeting keyword “clothing” causes presentation ofan ad to a user and the user subsequently selects (i.e., clicks) thepresented ad, the keyword “clothing” can be associated with the userselection. If the user subsequently completes a conversion, then theselection of the ad is identified as an assist click for the conversion.In this example, the keyword “clothing” is associated with an assistclick for the conversion by causing presentation of an ad that wasselected during the conversion cycle for the conversion. Thus, a keywordidentifier that represents the keyword can be associated with theconversion (e.g., stored at a memory location assigned to the conversionand/or being indexed according to the conversion identifier for theconversion) as an assist click keyword (i.e., a keyword that causedpresentation of a content item that was selected during a conversioncycle).

User interactions and/or conversions can be associated with one or moread groups and/or ad campaigns based on the associations of the userinteractions and/or conversions with targeting keywords that belong tothe respective ad groups and/or ad campaigns. For example, if aparticular targeting keyword is an assist click keyword for aconversion, the assist click can also be associated with the ad groupand/or ad campaign to which the particular targeting keyword belongs.

A conversion is selected from the user interaction data (704). In someimplementations, the conversion can be selected in response to receivinga request, from an advertiser, for an assisted conversions report for acontent distribution campaign (e.g., ad campaign). For example, anadvertiser may access a user interface similar to that described withreference to FIG. 7B, and request an assisted conversions report thatspecifies an assisted conversion measure for a particular ad campaign,particular ad group, or particular targeting keyword. In turn,conversions for computing the report can be selected.

In other implementations, the conversion can be selected as part of aperiodic assisted conversions report update. For example, an assistedconversions report can be generated and, in turn, updated periodicallyto provide an advertiser with an updated assisted conversions reportupon demand. Pre-computing the performance reports can provide fasteraccess to the report in response to the request. The description thatfollows describes selecting an individual conversion, but groups ofconversions can also be selected for processing.

The conversion is selected from user interaction data that areassociated with a particular advertiser. For example, user interactiondata associated with TPC can include data representing multipleconversions. In this example, one or more of the conversions can beselected from among the multiple conversions that are included in theuser interaction data for TPC. Conversions can be selected randomly, inan order in which they are indexed in the user interaction data, or inanother predetermined order. For example, the user interaction data caninclude a set of conversions that each has a unique conversionidentifier that uniquely identifies the conversion with which theidentifier is associated. In this example, the conversions can beselected in sequential order of the conversion identifiers. Eachconversion can also have a timestamp that specifies a time at which theconversion occurred, and the timestamp can be used to select theconversions in chronological order in which the conversions occurred.

Assist keywords that caused presentation of content items during aconversion cycle for the conversion are identified (706). Assistkeywords that are associated with the conversion, either as assist clickkeywords or assist impression keywords can be selected from the userinteraction data. For example, the user interaction data can includedata specifying a first and a second targeting keyword that respectivelycaused presentation of two content items during the conversion cycle forthe selected conversion, and that the user selected the second contentitem but not the first content item. In this example, the firsttargeting keyword is an assist impression keyword for the conversion andcan be selected from the user interaction data.

In some implementations, targeting keywords that caused presentation ofcontent items that are presented and/or selected prior to the last clickfor the conversion cycle are selected, and targeting keywords that areassociated with the last click of the conversion cycle (i.e., a keywordthat caused presentation of the last clicked content item) are notselected. Continuing with the example above, the second targetingkeyword will not be selected in these implementations because the secondtargeting keyword caused presentation of the content item that was lastselected prior to the conversion. Thus, the second targeting keyword isa last click keyword for the conversion rather than an assist impressionkeyword or an assist click keyword for the conversion.

A determination is made as to whether there are more conversions toprocess (708). If there are additional conversions to process, theprocess 700 returns to stage 704 of selecting a conversion from the userinteraction data. If it is determined that there are no remainingconversions to process, the process 700 proceeds to stage 710.

A quantity of conversions with which each assist keyword is associatedis determined (710). The quantity of conversions with which each assistkeyword is associated is referred to an assisted conversion count forthe assist keyword. In some implementations, the assisted conversioncount for each assist keyword is the total quantity of conversions forwhich the assist keyword caused presentation of at least one contentitem during the conversion cycles for the conversions. For example, eachtime the process 700 repeats stages 704 through 708 for a differentconversion, a counter associated with a targeting keyword can beincremented in response to determining that the targeting keyword isidentified at stage 706 as an assist keyword for the conversion. In thisexample, a final value of each counter will be the assisted conversioncounts for each respective assist keyword.

The assisted conversion count described above includes both the quantityof conversions for which the assist keyword was an assist click keywordand a quantity of conversions for which the assist keyword was an assistimpression keyword. In some implementations, a quantity of clickassisted conversions (i.e., conversions for which the assist keyword wasan assist click keyword) and a quantity of impression assistedconversions (i.e., conversions for which the assist keyword was anassist impression keyword) is determined separately for each assistkeyword. For example, assist keywords can be associated with separatecounters that track click assisted conversions and impression assistedconversions for the assist keyword. In this example, the assistedconversion counter can be incremented once for each conversion for whichthe assist keyword is associated with an assist click, but is notassociated with a last click, And the impression assisted conversioncounter can be incremented once for each conversion for which the assistkeyword is associated with an assist impression, but is not associatedwith an assist click or a last click.

In some implementations, each assist keyword can be credited with atmost one conversion count for each conversion irrespective of how manytimes the assist keyword is identified as an assist click keyword and/oran assist impression keyword for the conversion. For example, assumethat during a conversion cycle for a conversion, a particular assistkeyword causes presentation of five content items, three of which areselected by the user. In this example, the assist keyword can becredited with one assisted conversion, even though the assist keywordcaused presentation and selection of multiple content items.

An assisted conversion measure is computed (712). As described above, anassisted conversion is a conversion for which one or more assist userinteractions occur during a conversion cycle for the conversion.Assisted conversions can include, for example, conversions for which atleast one content item was presented to a converting user and zero ormore of the presented content items were selected by the converting userprior to a last click for the conversion. Assisted conversion measurescan be computed for each assist keyword that is identified as describedabove. The assisted conversion measures can also be computed on aper-advertiser, per content item, per-advertising campaign, and per-adgroup basis. For example, assisted conversion measures for an ad groupcan be computed using the assisted conversion measures for each oftargeting keywords that belong to the ad group. The assisted conversionmeasures can also be computed for other attributes of an advertisingcampaign.

In some implementations, the assisted conversion measures can include atotal quantity of conversions for which assist clicks and/or assistimpressions occurred during the conversion cycles for the conversions.For example, a quantity of click assisted conversions is a quantity ofconversions for which at least one content item is selected prior to alast click for the conversions. As another example, a first ad ispresented twice, followed by a presentation and selection of a secondad. A third ad is then presented and selected, followed by a conversion.The first and second ads are identified as assisting the conversionwhile the third ad is not identified as assisting the conversion sincethe third ad is associated with only the last click. In this example,the first ad can be associated with an impression assisted conversionand the second ad can be associated with a click assisted conversion. Insome implementations, if a content item is presented and not selectedand subsequently presented and selected in the same conversion cycleprior to a last click, the content item is associated with a clickassisted conversion and not an impression assisted conversion (e.g., theconversion is only counted once with respect to the content item).

The assisted conversion measures can also include an average quantity ofassisted conversions, click assisted conversions, or impression assistedconversions. Additionally, other statistical measures (e.g., standarddeviation, mode, median) and distributions of assisted conversions,click assisted conversions, and/or impression assisted conversions canbe generated. The discussion below describes several example measures ofuser interactions that can be computed, but other measures can becomputed.

In some implementations, computing an assisted conversion measureincludes computing a ratio of the quantity of assisted conversions,click assisted conversions, or impression assisted conversions relativeto the quantity of last clicks or conversions for an advertiser. Inthese implementations, the quantity of assisted conversions (orclick/impression assisted conversions) for the targeting keyword can bedivided by the quantity of last clicks for the targeting keyword tocompute the ratio of assisted conversions to last clicks for thetargeting keyword. For example, if the total quantity of last clicks fora particular keyword is 844 and the total quantity of assistedconversions for the particular keyword is 112, the ratio of assistedconversions to last clicks for the particular keyword is 0.13. Asanother example, if the total quantity of conversions for an ad campaignis 931 and the total quantity of click assisted conversions for the adcampaign is 274, the ratio of click assisted conversions to totalconversions for the ad campaign is 0.29.

In some implementations, a request for an assisted conversions reportthat specifies an assisted conversion measure for a content distributioncampaign is received (714). The request can be received, for example,from an advertiser that accesses a user interface, such as the userinterface described with reference to FIG. 7B, and selects a userinterface element that causes the data representing the request to besubmitted. The request can specify, for example, a report period for theassisted conversions report. For example, if the specified report periodis a one month period, then the assisted conversions report specifyassisted conversion measures for conversions that occurred during theone month period.

In response to the request, data representing one or more computedassisted conversion measures is presented to a user (716). For example,data that causes presentation of the total quantity of click assistedconversions for a one or more assist keywords associated with anadvertiser can be provided to a user device. As another example, datathat causes presentation of a web page that specifies the ratio ofassisted conversions to last clicks for one or more ad campaigns can beprovided to a user device. The data can be presented to the user, forexample, in the form of a graph, chart, table, or spread sheet. The datacan be presented as part of a webpage, or exported as a file (e.g., atext file, word processing file, spread sheet, or database file).

FIG. 7B is a screen shot of a conversion report user interface 720 forproviding assisted conversion measures (e.g., click assisted conversionand impression assisted conversion measures) are provided. The userinterface 720 can be presented, for example, in a web browserapplication that displays web pages and other resources that make upwebsites, (such as the websites 104 of FIG. 1). The conversion reportuser interface 720 includes a header 721 for indicating information thatis displayed in the conversion report user interface 720. In thisexample, the header 721 indicates that information associated with a webpage titled “MyAdvertisingAccount” is being presented and that thepresented information more specifically relates to assisted conversions.

The conversion report user interface 720 includes an assisted conversionreport 722 for presenting assisted conversion measures. The assistedconversion report 722 includes a number of selection controls (describedin more detail below) that enable a user to specify parameters for theassisted conversion measures. The assisted conversion report 722 can bepresented, for example, in a web page received from a web server overthe Internet. As another example, referring to FIG. 1, the assistedconversion report 722 can be presented with another resource 105 that isreceived over the network 102 by one of the user devices 106.

The conversion report user interface 720 includes an address bar 750that displays a URL for the assisted conversion report 722. In someimplementations, the user can access the assisted conversion report 722by entering the URL into the address bar 750. The user interface 720further includes a scroll bar 754 that enables the user to scrollthrough content presented on the assisted conversion report 722.

The assisted conversion report 722 includes a date range selectioncontrol 724 that enables the user to specify a date range for theassisted conversion measures that are presented in the assistedconversion report 722. The date range specified in the date rangeselection control 724 can be used to select data with which the assistedconversion measures that are presented in the assisted conversion report722 are computed. For example, the date range can be used to select userinteraction data that are associated with conversions that occurredwithin the specified date range, user interaction data that areassociated with conversion cycles where the entire conversion cycleoccurred during the specified date range, or user interaction data thatare associated with conversion cycles where any portion of theconversion cycle occurred during the specified date range. For example,user interaction data associated with a conversion cycle for which onlya first assist impression occurred during the specified date range canbe included in the information presented on the assisted conversionreport 722.

The assisted conversion report 722 includes active links 725 and 726that enable the user to specify a type of assisted conversion report topresent. In the example shown, selection of the link 725 causesinformation for a click assisted conversion report to be presented. Forexample, selection of the link 725 can cause submission of a request forthe click assisted conversion report. In response to the selection ofthe link 725, data that causes presentation of measures ofclick-assisted conversions and related data can be provided. Themeasures of click-assisted conversions can include, for example, a totalquantity of click-assisted conversions, an average quantity of clickassisted conversions, and other statistical measures of click assistedconversions.

Selecting the link 726 causes information for an impression assistedconversion report to be presented. For example, selection of the link726 can cause submission of a request for the impression assistedconversion report. In response to the selection of the link 726, datathat causes presentation of measures of impression-assisted conversionsand related data can be provided to the user device. The measures ofimpression-assisted conversions can include, for example, a totalquantity of impression-assisted conversions, an average quantity ofimpression-assisted conversions, and other statistical measures ofimpression-assisted conversions.

In the example shown, the link 725 is bold to indicate that the link 725is currently selected and that a click assisted conversions report iscurrently being presented by the assisted conversion report 722. Theclick assisted conversion report provides measures of click assistedconversions, where the conversions were completed during a date range ofMar. 16, 2010 to Apr. 15, 2010. Thus, user interaction data that areassociated with conversions that were completed during the specifieddate range were used to compute the measures of click-assistedconversions.

The click assisted conversion report provides measures of click-assistedconversions for a set of ad campaigns 732. The list of ad campaigns 732can be, for example, ad campaigns that each include one or more sets oftargeting keywords that control distribution of advertisements for aparticular advertiser. For example, a beach vacation ad campaign 737 caninclude the targeting keywords “beach,” “swimming,” “surfing,” “familyvacation,” “sun,” and “relaxing vacation.” Thus, when any of thesetargeting keywords are matched, as described above, one or more adsincluded in the ad campaign can be presented. As another example, a skivacation ad campaign can include the targeting keywords “skiing,”“winter vacation,” “Colorado vacation,” “snowboarding,” “snow vacation,”and “winter weekend getaway.”

In the example shown, the click assisted conversion report presentsclick assisted conversion statistics for each campaign in the set of thead campaigns 732. In some implementations, in place of or in addition topresentation of the set of ad campaigns 732, the click assistedconversion report can present a set of targeting keywords (e.g., the setof keywords 632 shown in FIG. 6B) associated with an advertiser and/orad campaign and the click assisted conversion statistics for eachtargeting keyword in the set of targeting keywords.

The click assisted conversion report can also present references (i.e.,advertisement identifiers) for advertisements in a set of advertisementsthat are distributed for an advertiser and/or by ad campaigns andpresent click assisted conversion statistics for each of theadvertisements. In some implementations, the click assisted conversionreport can include a list of products sold by the advertiser and presentclick assisted conversion statistics for ads and/or targeting keywordsthat are associated with each product.

The click assisted conversion report further includes a set of clickassisted conversion totals 734 indicating a total quantity of clickassisted conversions that are associated with each ad campaign 732. Theclick assisted conversion totals indicate that the beach vacation adcampaign 737 is associated with 712 click assisted conversions. Thus,the keywords that are included in the beach vacation ad campaignprovided at least one assist click for each of 712 conversions thatoccurred during the specified period. In some implementations, the text“beach vacation” is associated with a hyper-link that upon selectionrequests presentation of the keywords that are included in the “beachvacation” ad campaign and the quantity of assisted click conversions foreach of the keywords.

The assisted conversion report 722 includes a bar graph 736 thatspecifies click assisted conversion percentages for each of the adcampaigns 732 in relation to the total quantity of click assistedconversions for the specified date range. The bar graph 736 visuallyindicates the percentage of the total click assisted conversions thatare attributed to each ad campaign 732 for the specified period.Therefore, the user can visually inspect the percentages and identifythe relative contributions of each ad campaign 732 from a click assistedconversion perspective. For example, the user can readily identify thatthe beach vacation ad campaign is associated with more click assistedconversions than a cruise ad campaign since the bar at row 738 is largerthan the bar at row 740. In some implementations, selecting the link 726causes the bar graph 736 to present impression assisted conversionpercentage information. In some implementations, selecting a link forall assisted conversions causes bar graph 736 to present percentageinformation for all assisted conversions.

In some implementations, additional assisted conversion measures can bepresented on the click assisted conversion report. For example, theclick assisted conversion report includes text 742 indicating that thetotal quantity of click assisted conversions for the advertiser over thespecified date range is 1,909 click assisted conversions. As anotherexample, a ratio (e.g., presented as a percentage) of click assistedconversions relative to a quantity of last clicks for each ad campaign732 can be presented. As another example, a ratio (e.g., presented as apercentage) of impression assisted conversions relative to a quantity oflast clicks for each ad campaign 732 can be presented. As yet anotherexample, a ratio (e.g., presented as a percentage) of all assistedconversions relative to a total quantity of last clicks for each adcampaign 732 can be presented.

In some implementations, the assisted conversion report 722 includes alink that upon user selection generates a co-presentation request thatrequests presentation of both a click assisted conversion report and animpression assisted conversion report (i.e., all assisted conversions)in a same user interface. For example, in response to theco-presentation request, the click assisted conversion report and theimpression assisted conversion report can both be presented in the userinterface 720.

The assisted conversion report 722 includes a conversion type selectioncontrol 728 that enables the user to specify one or more types ofconversions for which the assisted conversion report includes relatedinformation. In some implementations, the conversion type selectioncontrol 728 can be a drop down menu having user selectable options. Inother implementations, the conversion type selection control can be ahypertext link or another user interface control element (e.g., a textbox, check box, or radio button). In the example shown, the conversiontype selection control 728 indicates that the assisted conversion reportincludes information for “all” types of conversions is presented on theassisted conversion report 722.

Conversion reports can be generated for each of many different types ofconversions. In some implementations, each of the following user actionscan be identified as a different conversion type: signing up for amailing list, making a purchase, creating a user profile, making areservation, requesting presentation of a video, requesting presentationof audio, downloading one or more files, installing one or more programson the user device, or providing specified information, and each aseparate path length report can be generated for each of these differentconversion types.

Each advertiser can specify its own set of conversion types and the useractions that constitute a conversion for each conversion type. Forexample, an on-line gaming company can identify three types ofconversions. The first conversion type can be identified as aninformation conversion that is completed when a mailing address(physical or electronic) is received from a user. The second conversiontype can be identified as a download conversion that is completed when auser downloads and/or installs a trial version of an on-line game. Thethird conversion type can be identified as a purchase conversion that iscompleted when a user purchases an on-line game. In this example, theconversion type selection control 728 enables allows the user (i.e., theadvertiser) to request presentation of an assisted conversion reportthat specifies assisted conversion measures for purchase conversionsand/or an assisted conversion report that specifies assisted conversionmeasures for other types of conversions (e.g., information conversionsor download conversions).

In some implementations, the assisted conversion report 722 canseparately specify, in a single user interface display, assistedconversion measures for more than one conversion type. Continuing withthe example above, in response to a request for an assisted conversionreport that includes separate assisted conversion measures for purchaseconversions and download conversions, data can be provided that causepresentation of the assisted conversion measures for the purchaseconversions at a display location that is adjacent to the displaylocation of the assisted conversion measures for the downloadconversions. Thus, a user can directly compare the measures assistedconversions for purchase conversions to the assisted conversion measuresfor the download conversions.

The assisted conversion report 722 includes an analysis type selectioncontrol 730 to allows the user to select from among multiple types ofconversion path performance measures (similar to the analysis typeselection control 630 shown in FIG. 6B). In the example shown, theanalysis type selection control 730 indicates that information currentlypresented on the assisted conversion report 722 relates to assistedconversions. Other types of conversion path performance measures caninclude time lag measures, path length measures, last click measures,first click measures, assist user interaction measures, and otherconversion path performance measures. In some implementations, theanalysis type selection control 730 can be a drop down menu having userselectable options.

In some implementations, the assisted conversion measures can bepresented in additional or alternative formats. For example, a pie graphsimilar to that shown in FIG. 3C can be presented showing the percentageof total click assisted conversions for each ad campaign 732. As anotherexample, a line graph showing the total quantity of click assistedconversions or impression assisted conversions for unique time periods(e.g., days or hours) over the course of the specified date range can beincluded on the assisted conversion report 722. As another example, theassisted conversion report 722 can include a table listing assistedconversion measures.

The assisted conversion report 722 includes a display type selectioncontrol 748 to allow the user to select one or more formats forpresenting assisted conversion measures on the assisted conversionreport 722. In the example shown, a selection control 749 is selected.The selection control 749 is used to select a bar graph format forpresentation of some or all of the assisted conversion measurespresented in or with the assisted conversion report 722. In the exampleshown, the display type selection control 748 additionally includesoptions for table, pie chart, and relative comparison chart formats.

The assisted conversion report 722 includes an export control 752 thatenables the user to export some or all of the assisted conversionmeasures that are computed and/or displayed in the assisted conversionreport 722. In some implementations, the export control 752 enables theuser to specify an export format. For example, the user can export themeasures as a spread sheet file. As another example, the user can exportthe measures in a word processing document.

Cross-Drill Measures

FIG. 8A is an example process 800 for computing user interactionmeasures for one or more targeting keywords. The one or more targetingkeywords (referred to as candidate keywords) can be, for example,targeting keywords that are associated with a particular userinteraction with a candidate content item (i.e., a content itemassociated with the conversion and with which the particular userinteraction occurred). As described in more detail below, the particularuser action can be a first click of a first clicked content item, a lastclick of a last clicked content item, or another user interaction with aspecified content item. Multiple user interaction measures can besequentially computed for a set of conversions, and each subsequentcomputation can be performed using the user interaction data for theconversions that were included in the previous user interaction measure.Thus, each sequential computation “drills down” into the data that wasused for the previous computation.

Performing sequential computations with user interaction data forconversions that were included in the previous user interaction measureenable an advertiser to filter data using criteria of a first report andthen viewing other reports that are generated using the filtered data.For example, conversion path performance measures relating to assistclicks for various keywords for an advertiser can be displayed as shownin FIG. 6A. A user can select one of the keywords 632 and select asecondary analysis type. If the user selects a secondary analysis typeof last clicks, keywords that caused content items associated with lastclicks to be presented for all of the conversion paths for which theselected keyword was associated with an assist click can be displayed.As another example, a list of last clicked keywords (i.e., keywords thatcaused presentation of the last clicked content item for a conversionpath) for conversions associated with an advertiser to be presented canbe displayed.

A user can select a keyword of “flights” from the list of keywords andselect a secondary analysis type of “assisted conversions.” A list ofkeywords that caused content items associated with assisted conversionsto be presented for conversion paths for which the selected keyword“flights” caused the last clicked content item to be presented can bedisplayed to the user. In other words, each of the displayed lastclicked keywords in this example is associated with an assistedconversion where the last click for each assisted conversion isassociated with the selected keyword “flights.” As yet another example,a user can select a beach vacation ad campaign from a list of adcampaigns associated with last clicks and select a drill down measure of“top paths” to cause a list of conversion paths where the last click ofthe conversion path is associated with the selected beach vacation adcampaign to be displayed.

The process 800 is a process by which user interaction data for userinteractions with content items is obtained, a request for a firstmeasure of user interactions for one or more targeting keywords isreceived, and a first set of user interactions data are selected inresponse to the received request. In some implementations, the selecteduser interaction data specifies conversions that are associated with theone or more targeting keywords and further specifies corresponding userinteractions for the conversions. In turn, a first measure of userinteractions for the one or more targeting keywords is computed.Subsequently, a request for a second measure of user interactions isreceived and a second measure of user interactions is computed.

The process 800 is described below with reference to advertisingcampaigns that control distribution of advertisements in an onlineenvironment. However, the process 800 can also be used with othercontent distribution campaigns that control distribution of othercontent (e.g., video, audio, game, or other content).

The process 800 can be performed, for example, by the performanceanalysis apparatus 120 shown in FIG. 1. In some implementations, theperformance analysis apparatus 120 is a data processing apparatus thatincludes one or more processors that are configured to perform actionsof the process 800. In other implementations, a computer readable mediumcan include instructions that, when executed by a computer, cause thecomputer to perform actions of the process 800. The stages shown in FIG.8A are illustrative of a set of stages that can be used to perform theprocess 800. Additional or fewer stages may be used, or stages may notbe used at all.

User interaction data specifying user interactions with content itemsare obtained (802). As described above, the user interaction data canrepresent user interactions that can include presentation of a contentitem (i.e., an impression) to a user, selection (i.e., a click) of acontent item by a user, and presentation of all or a portion of an audioand/or video content item to the user. In some implementations, the userinteraction data are collected over a specified time period. Forexample, user interaction data can be collected for all userinteractions that have occurred over the last two months.

In some implementations, the obtained user interaction data specify userinteractions that are associated with a particular advertiser, aparticular conversion, a particular targeting keyword, and/or contentitems with which the interaction occurred, as described above. Forexample, impressions and/or selections of content items that occurredwith respect to a user prior to a conversion by the user can beidentified as assist impressions, assist clicks, and last clicks asdescribed above with reference to FIG. 6A.

A conversions and/or user interactions with a content item that ispresented during a conversion cycle for the conversion can also beassociated with one or more keywords that caused presentation of thecontent item. User interactions and/or conversions can also beassociated with one or more ad groups and/or ad campaigns based on theassociations of the user interactions and/or conversions with targetingkeywords that belong to the respective ad groups and/or ad campaigns, asdescribed above.

A request for a first measure of user interactions (“first measure”) forone or more targeting keywords is received (804). The request for thefirst measure can specify that the first measure be computed for a firstset of user interface data. In some implementations, the first set ofuser interactions can include user interactions that are associated withconversions for which one or more targeting keywords caused presentationof content items with which a particular user interaction occurred. Asdescribed above, the particular user interaction can be a first click ofa first clicked advertisement, a last click of a last clickedadvertisement, or another specified user interaction. In someimplementations, the particular user action can be required to haveoccurred prior to a last click for a conversion. The content items withwhich the particular user interaction occurred are referred to generallyas “candidate content items.”

A user can access a user interface to specify parameters for which oneor more measures of user interactions are to be computed. For example,the user can specify a group of targeting keywords (e.g., keywords inone or more ad groups and/or one or more ad campaigns) for which themeasures of user interactions are to be computed. Additionally, the usercan access the user interface to specify that the first measure to becomputed is an analysis of “first click” (or “last click” analysis) userinteractions. Other parameters that the user can specify can include,for example, a date or time range of user interaction data, a conversiontype for the user interaction data, an assist user interaction type forthe user interaction data (e.g., click, impression, or all), an analysistype for the first measure, a targeting keyword grouping indication, anda display format for the first measure.

A first click conversion analysis can provide, for each keyword, aquantity of conversions for which the keyword caused presentation of acontent item that was the first content item selected by a user during aconversion cycle for a conversion. A last click conversion analysis canprovide, for each keyword, a quantity of conversions for which thekeyword is a last clicked keyword (i.e., caused presentation of acontent item that was the last content item selected by a user prior tothe conversion).

The last click conversion analysis can additionally provide a ratio oflast clicks to first clicks for a keyword, a ratio of last clicks forthe keyword to last clicks for all keywords, a ratio of last clicks toassist clicks for a keyword, a ratio of last clicks to assistedconversions for a keyword, or other statistics related to last clicks ofcontent items presented as part of conversion cycles in response a matchfor the keyword. Additionally, the last click conversion analysis caninclude other statistical measures (e.g., standard deviation, mode,median) and distributions of last clicks.

In some implementations, the request for a first measure includes arequest for a first impression conversion analysis or a last impressionconversion analysis that respectively specifies a quantity ofconversions for which each targeting keyword causes presentation of afirst or last presented content item. Other types of first or lastimpression conversion analyses can include similar calculations asdescribed above for first and last click conversion analyses. Forexample, a separate first or last impression conversion analysis can beseparately performed for each of one or more different conversion types,in a manner similar to that described with reference to FIG. 7B.

In other implementations, the request for a first measure can be arequest for a measure of assist user interactions for a targetingkeyword, a measure of time lag for a targeting keyword, a measure ofassisted conversions for a targeting keyword, or a measure of pathlength for a targeting keyword.

In some implementations, requests for measures of user interactions areautomatically submitted when the user submits or edits a parameter. Insome implementations, the user submits requests for measures of userinteractions by selecting a submit (i.e., enter) control. A submittedrequest for a measure of user interactions can be received by a systemcapable of computing or identifying conversion path performancemeasures. For example, referring to FIG. 1, the performance analysisapparatus 120 can receive the first measure.

The one or more targeting keywords can be keywords associated with anadvertiser, an ad campaign, an ad group, a content item, or one or morespecified products. For example, the request can specify userinteractions for targeting keywords associated with a weekend getaway adcampaign. As another example, the one or more targeting keywords can beassociated with a single content item.

A first set of user interaction data are selected (806). In someimplementations, the first set of user interaction data specifiesconversions that are associated with the particular user interactionwith the candidate content item and further specifies corresponding userinteractions for these conversions. For example, the selected userinteraction data can be data associated with conversions for which theone or more targeting keywords caused content items associated withassist clicks or assist impressions for the conversions to be presented.As another example, the selected user interaction data can specifyconversion paths of conversions for which the one or more targetingkeywords caused content items for the conversion paths to be presented(e.g., the keywords are associated with assist clicks, assistimpressions, and last clicks for the conversion paths).

For example, the first set of user interaction data may include datarepresenting a conversion for which a particular targeting keywordcaused presentation of a first clicked content item. In this example,the user interaction data associated with this conversion can includedata that represents the particular targeting keyword as the firstkeyword (K1) in the conversion path for the conversion. The userinteraction data can also include data specifying the remaining keywords(K2-Kn) that respectively caused sequential presentation of contentitems over the conversion cycle for the conversion. Thus, the first setof user interaction data can specify that the conversion path of for theconversion includes K1-Kn in sequential order.

In some implementations, the first set of user interaction data alsoindicates, for the conversions that are associated with the particularuser interaction with the candidate content item, assist clicks for theconversions, assist impressions for the conversions, last clicks for theconversions, and targeting keywords, content items, ad campaigns, or adgroups associated with the conversions, assist clicks, assistimpressions, and/or last clicks. For example, if the request received atstage 804 includes an indication of a keyword “running shoes,” the firstset of user interaction data can include information for all conversionpaths that include content items presented in response to a match forthe keyword “running shoes.”

As another example, when the request specifies that the first measure isto be computed for an ad campaign named “Generic products” using userinteraction data for conversions for which keywords in the ad campaign“Generic” caused presentation of the last clicked advertisement, thefirst set of user interaction data can include information for allconversion paths where keywords from the ad campaign “Generic” causedpresentation of last clicked content items. In this example, the firstset of user interaction data can include data specifying each contentitem included in the conversion paths, keywords that, at least in part,caused the content items to be presented, and ad groups that include thekeywords. The first set of user interaction data can additionallyspecify whether each content item is associated with an assist click, anassist impression, or a last click for a particular conversion path.

The first measure for the one or more targeting keywords is computed(808). The first measure can be computed on a per-advertiser,per-content item, per-advertising campaign, per-ad group, or per-keywordbasis. In some implementations, the computed first measure can specify aquantity of conversions for which the targeting keywords causedpresentation of the candidate content item.

For example, if the request received at stage 804 indicates that lastclick related measures for the targeting keywords are requested, thefirst measure can be a total quantity of conversions for which eachtargeting keyword caused a last clicked content item to be presented.

In other implementations, the first measure can specify a quantity ofassist clicks, assist impressions, total assist user interactions, clickassisted conversions, impression assisted conversions, total assistedconversions, last clicks, last impressions, first clicks, firstimpressions, a time lag measure, a path length measure, user interactionpaths, any combination of these measures, or any previously describedconversion path performance measures associated with the one or moretargeting keywords (or ad campaigns, ad groups, etc.).

For example, if the request received at stage 804 indicates that clickassisted conversions measures for the targeting keywords are requested,the first measure can be a total quantity of conversions for which eachtargeting keyword (or ad campaign) caused a content item to be presentedin the conversion paths of the conversions prior to the last click forthe conversions where the presented content item was selected by a user.As yet another example, average path lengths for the indicated keywordscan be computed. As yet another example, paths that include theindicated keywords can be computed and a total quantity of times thateach path lead to a conversion can be computed.

Additional conversion path performance measures that can be included inthe computed first measure can include ratios of assist clicks to lastclicks, assist impressions to last clicks, assist user interactions tolast clicks, assisted conversions to last clicks, click assistedconversion to last clicks, impression assisted conversions to lastclicks, assist interactions for a keyword to total assist interactions,or assisted conversions for a keyword to total assisted conversions. Forexample a ratio of assist clicks associated with each targeting keyword(or ad campaign) indicated in the request received at stage 804 to atotal quantity of assist clicks for all targeting keywords can becomputed. As another example, a ratio of assisted conversions associatedwith an ad campaign to last clicks associated with the ad campaign canbe computed.

A request for a second measure of user interactions (“second measure”)is received (810). In some implementations, the request for the secondmeasure is a request that temporally and/or sequentially follows therequest for the first measure. For example, the request for the secondmeasure can be a request for a quantity of assist clicks, assistimpressions, total assist user interactions, click assisted conversions,impression assisted conversions, total assisted conversions, lastclicks, last impressions, first clicks, last clicks, first impressions,a time lag measure, a path length measure, user interaction paths, anycombination of these measures, or any previously described conversionpath performance measures associated with the one or more targetingkeywords (or ad campaigns, ad groups, etc.). This request cansequentially follow, for example, a previously received first requestfor a first click analysis or a last click analysis.

The request for a second measures of user interactions can be a requestfor more specific measures (relative to those provided by the firstmeasure) of user interactions for the conversions, targeting keywords,and/or content items that are included in the first set of userinteraction data. As described in more detail below, the parameters thatwere specified for the first measure can be used as conditions that mustbe satisfied for a conversion to be analyzed. In some implementations,the conversions that were included in the first measures (or a subset ofthe conversions that were included in the first measures) can be theonly conversions that are eligible to be analyzed for the secondmeasure. For example, following computation of first measures of userinteractions, a particular keyword might be identified as the firstclick keyword for 10 conversions. When the request for the secondmeasures is received, the second measure can be computed using those 10conversions.

In some implementations the request for the second measure of userinteractions is received in response to presentation of the firstmeasure. For example, the first measure of user interaction computed atstage 808 can be a last click analysis for one or more ad campaigns thateach include a group of targeting keywords. The last click analysis caninclude a determination of a quantity of conversions for which targetingkeywords were last clicked keywords. The results of the last clickanalysis can be presented to a user (e.g., as part of a webpage, or in aspread sheet).

Following presentation of the results, the user can initiate a requestfor a second measure by selecting an ad campaign from the list of adcampaigns and/or selecting a second analysis type. For example, the usercan select an analysis type of path length in order to initiate arequest for a path length analysis (i.e., a request for a path lengthreport) of conversions for which a targeting keyword in the selected adcampaign was a last clicked keyword. The path length analysis caninclude, for example, data specifying path lengths for the conversionpaths for which a targeting keyword in the ad campaign was a lastclicked keyword.

As another example, a list of ad groups that each include multiplekeywords that are associated with first clicks for conversionsassociated with an indicated advertiser is presented to a user alongwith a first click analysis for the keywords including a quantity ofconversions for which the keywords in each ad group caused presentationof a first clicked advertisement. The user can select one of the adgroups to request more specific (e.g., more detailed or granular)analysis of the keywords that are included in the ad group. In responseto the selection of the ad group, the first measures for each of thetargeting keywords in the ad group can be presented. The user can thenrequest a top path report for the conversions that were used to performthe first click analysis. That is, the user request is a request for atop path analysis for conversion paths for which a targeting keyword inthe ad group caused presentation of the first clicked ad in theconversion path.

The second measure of user interactions is computed (812). The requestfor the second measure can include a request for further analysis ofinteractions for conversions that are associated with the particularuser interaction with the candidate content item. Therefore, the secondmeasure can be computed using user interaction data for conversions thatare associated with the particular user interaction with the candidatecontent item. For example, the request for the second measure canrequest analysis of the conversions that were included in the firstmeasure. As described above the second measure can be any of thepreviously described measures of user interactions or another measure ofuser interactions.

For example, assume that an advertiser first requested a last clickanalysis of its ad campaign “basketball.” In response to the request,user interaction data that specified targeting keywords for the“basketball” campaign as last click keywords are selected, and theselected user interaction data are used to compute measures of userinteractions for the “basketball” campaign.

In response to presentation of the measures of user interactions, theuser selects a second type of analysis to perform, which as describedabove, initiates a request for measure of user interactions for thesecond type of analysis. For example, the request can be for a pathlength analysis. In this example, rather than performing the path lengthanalysis on all available user interaction data, the second analysis isperformed using the user interaction data that indicate that targetingkeywords for the “basketball” campaign are last click keywords. Thus,the computation of path length measures are performed using userinteraction data that indicate that a targeting keyword in the“basketball” campaign is associated with a particular user interaction(i.e., a last click) with a candidate content item (i.e., a last clickedadvertisement). By performing the analysis using data that was selectedto compute the first measure, the second measure can provide moredetailed measures of user interactions for the conversions that wereincluded in the first measures of user interactions.

The second measure can be computed on a per-advertiser, per-contentitem, per-advertising campaign, per-ad group, or per-keyword basis. Themeasures of assisted conversions can also be computed for otherattributes of an advertising campaign (e.g., a per-product basis).

In some implementations, data that cause presentation of the firstmeasure and/or the second measure are provided to a user. For example,data that cause presentation of a last click analysis for an advertisingcampaign and/or a path length analysis for conversion paths ofconversions that were included in the last click analysis can beprovided. The data can be provided, for example, to a user device thatprocesses the data and, in turn, displays measures of user interactionsto the user. The measures of user interactions can be presented to theuser, for example, in the form of a graph, chart, table, or spreadsheet. The data can be presented as part of a webpage, or exported as afile (e.g., a text file, word processing file, spread sheet, or databasefile).

FIG. 8B shows a screen shot of an example user interface 820 forproviding measures of user interactions. The user interface 820 can bepresented in a browser that displays web pages and other resources fromvarious websites, (such as the websites 104 of FIG. 1). The userinterface 820 includes a header 821 for indicating information that isdisplayed in the user interface 820. In this example, the header 821indicates that information associated with a web page titled“MyAdvertisingAccount” is being presented and that the presentedinformation more specifically relates to a drill down analysis startingwith a last clicks analysis that is drilled down to an assistedconversions analysis. This can indicate that the information displayedin the user interface 820 includes an analysis of one or more assistedconversions measures for conversion paths having last clicks associatedwith an indicated keyword, ad campaign, ad group, or content item.

The user interface 820 includes a report area 822 for presentinginformation relating to conversion path performance measures and severalselection controls to allow a user to specify parameters for theconversion path performance measures. The report area 822 can be, forexample, a webpage received from a web server through the Internet.

The user interface 820 includes an address bar 850 that displays a URLfor the report area 822. In some implementations, the user can accessthe report area 822 by entering the URL into the address bar 850. Thebrowser further includes a scroll bar 854 to allow the user to scrollthrough content presented on the report area 822.

The report area 822 includes a page title 823 indicating that theinformation included on the report area 822 relates to assistedconversions for conversion paths having a last click keyword “bookflight.” In other words, the assisted conversion information presentedon the report area 822 relates only to conversion paths where thekeyword “book flight” caused a last clicked content item to bepresented. The information can be presented in the report area 822, forexample, in response to a user request for conversion paths having alast keyword “book flight.”

The report area 822 further includes a date range selection control 824to allow the user to specify a date range for conversion pathperformance measures to be presented on the report area 822. The daterange specified in the date range selection control 824 can limitinformation presented on the report area 822 to information relating toconversions that occurred within the specified date range, informationrelating to conversion cycles where the entire conversion cycle occurredduring the specified date range, or information relating to conversioncycles where any portion of the conversion cycle occurred during thespecified date range. For example, information relating to a conversioncycle where a first assist impression occurred during the specified daterange can be included in the information presented on the report area822.

The report area 822 includes links 825 and 826 that enable the user toselect a type of assisted conversion information to present. In theexample shown, selecting the link 825 causes information relating toclick assisted conversions to be presented. Selecting the link 826causes information relating to impression assisted conversions to bepresented. In the example shown, the link 825 is bold to indicate thatthe link 825 is currently selected and that information presented on thereport area 822 is related to click assisted conversions. In someimplementations, the report area 822 includes a link that allows theuser to view information relating to both click assisted conversions andimpression assisted conversions (i.e., all assisted conversions).

The report area 822 includes a conversion type selection control 828that enables the user to indicate one or more types of conversions tofor which to present related information. In the example shown, theconversion type selection control 828 indicates that informationrelating to all types of conversions is presented on the report area822. In some implementations, conversions can be divided into differenttypes. As described above, user actions that can be identified asdifferent conversion types can include, signing up for a mailing list,making a purchase, creating a user profile, making a reservation,watching a video, listening to an audio file, downloading one or morefiles, installing one or more programs, or providing information. Insome implementations, the conversion type selection control 828 can be adrop down menu having user selectable options.

The report area 822 includes an analysis type selection control 830 thatenables the user to select from among multiple types of initialconversion path performance measure analyses. In the example shown, theanalysis type selection control 830 indicates an initial analysis typeof “last clicks” is selected. Other types of conversion path performancemeasures can include time lag measures, path length measures, userinteraction path measures, assisted conversion measures, first clickmeasures, and assist user interaction measures. In some implementations,the analysis type selection control 830 can be a drop down menu havinguser selectable options.

The page includes a drill down analysis type control 831. The drill downanalysis type control 831 allows the user to select a secondary or“drill down” analysis type for information displayed on the report area822. The analysis type selection control 830 indication of “last clicks”indicates that an initial last click analysis is performed for a set ofkeywords (or ad campaigns, advertisers, ad groups, content items, etc.).A keyword from the set of keywords is then indicated (e.g., by beingselected by the user or entered into a text field in a previous view ofthe user interface 820 or a previously presented user interface). Theuser then indicates a secondary analysis type of “assisted conversions”using the drill down analysis type control 831. This indicates that theuser is requesting an analysis of assisted conversions for conversionpaths for which the last clicked content item was presented in responseto a match for the indicated keyword (the indicated keyword in thisexample being “book flight.”). Allowing the user to specify a secondaryanalysis can allow the user to readily view information relating tokeywords (or ad campaigns, ad groups, etc.) that have assistedconversions where the last click in the conversion is associated withthe keyword “book flight.” This can assist the user in identifying oneor more keywords that work well in conjunction with other keywords.

As mentioned above, the report area 822 includes conversion pathperformance measures as specified by the user using the selectioncontrols and links of the report area 822. In the example shown, thereport area 822 includes information relating to click assistedconversions associated with all conversion types for a date range ofMar. 16, 2010 to Apr. 15, 2010 for conversion paths for which the lastclicked content item was presented in response to a match for thekeyword “book flight.” The information presented on the report area 822includes a list of keywords 832. The list of keywords 832 can be, forexample, keywords that caused advertisements to be presented as part ofconversion paths for which the last clicked content items for theconversion paths were presented in response to a keyword of “bookflight” and the presented advertisements were subsequently selected by auser.

In the example shown, the report area 822 presents click assistedconversion statistics relating to each of the keywords 832. In someimplementations, in place of or in addition to the list of keywords 832,the report area 822 can include a list ad campaigns (e.g., the list ofad campaigns 732 shown in FIG. 6B) or ad groups associated with anadvertiser and show click assisted conversion statistics for each listedad campaign or ad group. In some implementations, the report area 822can include a list of advertisements associated with an advertiser or adcampaign and show click assisted conversion statistics for each listedadvertisement. In some implementations, the report area 822 can includea selection control that enables the user to select a category ofinformation for which to present statistics (e.g., keywords,advertisements, ad campaigns, ad groups, or products).

The report area 822 further includes a list of click assisted conversiontotals 834 indicating a total quantity of click assisted conversionsassociated with each keyword 832 for the date range specified in thedate range selection control 824 for conversion paths having the keywordof “book flight” associated with the last clicks for the conversionpaths. The displayed information can allow the user to readily identifykeywords that assist the last click keyword of “book flight.” Forexample, the click assisted conversion totals indicate that a keyword“flights” is associated with 120 click assisted conversions forconversion paths having the last click keyword of “book flight.”Similarly, a keyword “vacation” is associated with 68 click assistedconversions for conversion paths having the last click keyword of “bookflight.”

These values may be different from general assisted conversion measuresfor all conversion paths. For example, the keyword “vacation” may assistmore conversions than the keyword “flights” for all conversion paths,while the information displayed in the example indicates that thekeyword “flights” assists more conversions than the keyword “vacation”for the subset of conversion paths having the last click keyword “bookflight.” In some implementations, selecting the link 826 causes the listof click assisted conversion totals 834 to be replaced by a list ofimpression assisted conversion totals for conversion paths having thelast click keyword “book flight.”

The report area 822 includes a bar graph 836 that indicates clickassisted conversion percentages for each of the keywords 832 in relationto the total quantity of click assisted conversions for the subset ofconversion paths having the last click keyword “book flight.” The bargraph 836 visually indicates the percentage of click assistedconversions attributed to each keyword 832 in relation to the totalquantity of click assisted conversions for the subset of conversionpaths that enables the user to readily visually inspect the totals andidentify the relative contributions of each keyword 832 from a clickassisted conversion perspective for the subset of conversion paths. Forexample, the user can readily identify that the keywords “flights,”“travel,” “hotel” and “vacation” are associated with significantly moreclick assisted conversions for the subset of conversion paths than theremaining keywords 832 since the bars associated with the keywords“flights,” “travel,” “hotel” and “vacation” are significantly longerthan the bars associated with the remaining keywords 832. Thisinformation can assist the user identifying the most successful keywords832 for use in on-going and future advertising strategies. In someimplementations, selecting the link 826 causes the bar graph 836 topresent impression assisted conversion percentage information. In someimplementations, selecting a link for all assisted conversions causesbar graph 836 to present percentage information for all assistedconversions for the subset of conversion paths.

In some implementations, additional conversion path performance measurescan be presented on the report area 822. For example, the report area822 includes text 842 indicating that the total quantity of clickassisted conversions for conversion paths having a last click keyword of“book flight” over the specified date range is 365 click assistedconversions. As another example, the user can use the drill downanalysis type control 831 to select a different secondary analysis type,such as, assist user interactions, time lag, top paths, first clicks orpath length analyses. As another example, the user can indicate adifferent initial analysis type using the analysis type selectioncontrol 830 and drill down into the newly indicated analysis type usingthe drill down analysis type control 831.

In some implementations, the assisted conversion measures can bepresented in additional or alternative formats. For example, a pie graphsimilar to that shown in FIG. 3C can be presented showing the percentageof total click assisted conversions for each keyword 832. As anotherexample, a line graph showing the total quantity of click assistedconversions or impression assisted conversions for unique time periods(e.g., days or hours) over the course of the specified date range can beincluded on the report area 822. As another example, the report area 822can include a table listing conversion path performance measures.

The report area 822 includes a display type selection control 848 thatenables the user to select one or more formats for presenting conversionpath performance measures on the report area 822. In the example shown,a selection control 849 is selected. The selection control 849 is usedto select a bar graph format for presentation of some or all of theconversion path performance measures presented on the report area 822.In the example shown, the display type selection control 848additionally includes options for table, pie chart, and relativecomparison chart formats.

The report area 822 includes an export control 852 that allows the userto export some or all of the assisted conversion measures displayed onthe report area 822. In some implementations, the export control 852allows the user to specify an export format. For example, the user canexport the measures as a spread sheet file. As another example, the usercan export the measures in a word processing document

Multi-Channel Measures

FIG. 9A is an example of a user interface 920 for providing conversionpath performance measures that are computed from multiple sources ofuser interaction data. The user interface 920 can be, for example,presented by a user device executing a browser that is used to displayweb pages and other resources (such as the websites 104 of FIG. 1).Conversion path performance measures that can be provided by the userinterface 920 include time lag measures, path length measures, top pathanalysis, assist user interaction measures, assisted conversionmeasures, last click measures, first click measures, and any otherpreviously described measure or analysis as well as additionalperformance measures.

The user interaction data used to create the conversion path performancemeasures displayed by the user interface 920 can be drawn from multiplesources of user interaction data. Each of the multiple sources of userinteraction data can provide, for example, user interaction data thatdescribes user interactions with different types of content. Forexample, one of the sources of user interaction data can be anadvertisement management system (or a data store associated therewith)that obtains data representing user interactions with advertisementsthat are presented with search results, where the advertisements arepresented with the search results in response to targeting keywords forthe advertisements being matched. Similarly, one of the sources of userinteraction data can be a same or different advertisement managementsystem that obtains data representing user interactions withadvertisements that are presented on web pages that are provided bycontent publishers. Still another source of user interaction data can bean analytics apparatus that tracks user interactions with web pages thatare provided, for example, by an advertiser. The sources of userinteraction data can include one or more search engines, advertisementdistribution systems, e-mail distribution systems, other contentdistribution systems (e.g., streaming video distribution systems),content performance tracking systems, advertisers, and advertiseraffiliates.

The user interaction data can include, for example, information relatingto interactions (e.g., impressions, clicks, and other informationsubmission) associated with search results for one or more searchengines, paid search results associated with one or more search engines,e-mails (e.g., e-mail newsletters), banner ads, other paidadvertisements, advertiser affiliates (e.g., special offers for relatedproducts/services), video content, audio content, or web pagesassociated with an advertiser (e.g., pages of the advertiser's website).

The user interaction data can also include resource performance datathat specify one or more user interactions with a resource, such asinteractions with a web page, a widget (i.e., an interactive applicationthat is provided with a web page), or another online resource. Theresource for which the resource performance data are specified isreferred to below as a measured resource. The resource performance datacan be, for example, web page (or website) performance analyticinformation that can provide information specifying referring pages(i.e., web page from which visitors navigated to the measured resource),destination pages (i.e., web pages to which visitors navigated followingpresentation of the measured resource). The resource performance datacan also include data specifying user interactions with a resource overa specified period (e.g., a specified time, or a specified quantity ofinteractions, as well as types of interactions that occurred (i.e.,selections of content, downloads, data that was input).

The user interaction data can be analyzed to identify conversion pathsinvolving user interactions with some or all of the above mentionedforms of content. For example, a user can receive an e-mail newsletterand click on a link in the e-mail newsletter to be taken to anadvertiser's website. A banner ad for the advertiser is later presentedto the user, followed by a portion of a streaming video associated withthe advertiser being presented to the user. The user can then enter asearch query and be presented with a paid search result for theadvertiser. The user selects the paid search result and subsequentlyperforms an action that constitutes a conversion (e.g., makes apurchase). The conversion path for the conversion in this exampleincludes an assist click associated with the e-mail newsletter, anassist impression associated with the banner ad, an assist impressionassociated with the streaming video, and a last click associated withthe paid search result.

As another example, an identified conversion path can include an assistimpression associated with a special offer presented by an advertiseraffiliate, an assist click associated with a banner ad, an assistimpression associated with a streaming audio file, an assist clickassociated with an organic search result (i.e., a non-paid searchresult) for a first search engine, and a last click associated with apaid search result for a second search engine. In some implementations,conversion path performance measures are calculated using the identifiedconversion paths involving user interactions with content items fromeach of the multiple sources. For example, keywords associated with eachuser interaction can be identified and a quantity of assist clicks andassist impressions (collectively “assist user interactions”) associatedwith each keyword can be computed. In other implementations, theconversion path performance measures can be computed using conversionspaths for a set of user interactions with content items from a subset ofthe multiple sources that are available (i.e., using at least one sourcefewer than the total quantity of available sources).

As described above, information associated with user interactions with aweb page can be identified using collected analytics data. When a useraccesses a web page, or another resource, from a referring web page (orother resource), the referring web page (or other resource) for thatinteraction can be identified, for example, by a snippet of code that isassociated with the web page that is being accessed and/or based on aURL that is associated with the link that was selected. For example, auser can access an advertiser's website by selecting a link presented ona web page, for example, as part of a promotional offer by an affiliateof the advertiser. This link can be associated with a URL that includesdata (i.e., text) that uniquely identifies the resource from which theuse is navigating. For example, the linkhttp://www.example.com/homepage/%affiliate_identifier%promotion_(—)1 canspecify that the user navigated to the example.com web page from a webpage of the affiliate that is associated with the affiliate identifierquantity that is specified in the URL, and that the user was directed tothe example.com web page based on a selection of the link that isincluded in the promotional offer that is associated withpromotion_(—)1. The user interaction data for this interaction (i.e.,the selection of the link) can be stored in a database and used, asdescribed below, to compute conversion path performance measures.

As another example, a search results page can be identified as areferring resource that referred a user to the measured web page, forexample, by selection of an organic or paid search result. In additionto identifying the search engine (or search provider) that referred theuser to the measured web page, information, such as one or more searchterms that were used to generate the search results page that referredthe user to the measured web page can be specified, as well asinformation regarding whether the referral was from an organic searchresult, a paid search result, or a paid advertisement that was presentedwith the search results page. As described above, the URL that is usedto refer the user to the measured web page can include the informationdescribed above, as well as additional information.

In some implementations, pages of an advertiser's website can includetracking code for collecting analytics data. For example, tracking codecan be added to pages of the advertiser's website to track resourcesthat referred users to the website. The tracking code can also be usedto track user interactions among multiple web pages within a measuredwebsite. For example, the tracking code can log each user interactionwith the web pages and timestamp each user interaction. The trackingdata collected by the tracking code can be logged (i.e., saved to a datastore) and used to identify assist impressions and other userinteractions (assist clicks, last clicks, interaction paths, etc.)associated with the measured website.

In some implementations, cookies can be used to track user interactions.For example, a cookie can be used to identify and track when contentitems are requested by a user and user interactions with the contentitems (e.g., presentations and/or selections of a content item, contentdownloads, or information submissions). In some implementations,multiple cookies can be used to track user interactions with contentitems provided by different content distributors and/or to trackdifferent types of user interactions.

For example, a first cookie is used to track user interactions withsearch results provided by a first search engine while a second cookieis used to track user interactions with search results provided by asecond search engine. As another example, a first cookie is used totrack user interactions with streaming video files while a second cookieis used to track user interactions with banner ads displayed on webpages and a third cookie is used to track user interactions withe-mails. The information collected using the cookies can be combined toidentify conversion paths and compute conversion path performancemeasures. In some implementations, the information collected usingcookies is combined with other user interaction data and used toidentify conversion paths and compute conversion path performancemeasures.

In some implementations, collected user interaction data are associatedwith one or more user identifiers and/or one or more conversionidentifiers. For example, user interaction data relating to a firstconversion is associated with a first user identifier and/or a firstconversion identifier, and data relating to a second conversion isassociated with a second user identifier and/or a second conversionidentifier. Each user identifier uniquely represents a converting userfor a conversion, while each conversion identifier uniquely identifieseach conversion. For example, when a conversion is an online purchase ofa baseball glove, a user identifier for the user device and/or browserthat were used to purchase the baseball glove is identified, forexample, from a cookie that is stored on the user device. As anotherexample, a user identifier for a user device that was used to sign upfor a mailing list on a concert website can be identified. As describedabove, the user identifier can be a cookie obtained from the user devicethat was used to complete the conversion a hash of a user name, arandomly assigned user ID code, or another anonymized user identifier.

In some implementations, user interaction data are associated with auser by being stored with a user identifier that represents theconverting user and/or by being indexed according to the useridentifier. In some implementations, user interaction data are indexedaccording to conversion identifiers for conversions with which the userinteraction data are associated. For example, each conversion can beassociated with a unique conversion identifier, and user interactiondata associated with the conversion path for a conversion is indexedaccording to the conversion identifier for the conversion.

Still referring to FIG. 9, the user interface 920 includes a header 921for indicating information that is displayed in the user interface 920.In this example, the header 921 indicates that information associatedwith a website titled “MyAdvertisingAccount” is being presented and thatthe presented information more specifically relates to a top pathanalysis for an advertiser. The user interface 920 includes a reportarea 922 for presenting information relating to conversion pathperformance measures and several selection controls that enable a userto specify parameters for the conversion path performance measures. Thereport area 922 can be, for example, a webpage received from a webserver through the Internet.

The user interface 920 includes an address bar 950 that displays a URLfor the report area 922. In some implementations, the user can requestpresentation of a performance report in the report area 922 by enteringthe URL for a requested report into the address bar 950. The userinterface 920 further includes a scroll bar 954 that enables the user toscroll through content presented on the report area 922.

The report area 922 includes a page title 923 indicating that theinformation included on the report area 922 relates to a top pathanalysis of user interaction data obtained from multiple sources. Forexample, top conversion paths identified using user interaction dataassociated with multiple content item distribution sources can bedisplayed parting a portion of the report area 922. The user interactiondata can include user interactions with search results (both paid andunpaid), advertisements, e-mails, streaming video or audio content, orspecial offers presented by affiliates. The user interactions caninvolve content items provided by multiple content item providers.

The report area 922 includes a conversion type selection control 928that enables the user to indicate one or more types of conversions tofor which to present related information. In the example shown, theconversion type selection control 928 indicates that informationrelating to all types of conversions is presented on the report area922. In some implementations, conversions can be divided into differenttypes. As described above, user actions that can be identified asdifferent conversion types can include, signing up for a mailing list,making a purchase, creating a user profile, making a reservation, beingpresented with a video, being presented with an audio file, downloadingone or more files, installing one or more programs, or providinginformation. In some implementations, the conversion type selectioncontrol 928 can be a drop down menu having user selectable options.

The report area 922 includes an analysis type selection control 930 thatenables the user to select from among multiple types of conversion pathperformance analyses. In the example shown, the analysis type selectioncontrol 930 indicates an analysis type of “top paths” is selected. Othertypes of conversion path performance measures can include time lagmeasures, path length measures, user interaction path measures, assistedconversion measures, first click measures, last click measures, andassist user interaction measures as well as other performance measures.In some implementations, the analysis type selection control 930 can bea drop down menu having user selectable options.

The report area 922 includes interaction event type selection controls931 that enable the user to select one or more types of user interactionevent types (and/or sources of user interaction data with which togenerate conversion paths and/or compute conversion path performancemeasures. The interaction event type selection controls 931 includeoptions for selecting multiple search engines as sources of userinteraction data (search engine 1 and search engine 2). In someimplementations, the user interaction data collected for the searchengines includes organic search results (i.e., non-paid search results).In some implementations, the user interaction data collected for thesearch engines includes both paid and non-paid search results, which canbe tracked as described above. In some implementations, user interactiondata collected for the search engines includes data for userinteractions with advertisements displayed with search results and aretracked, for example, by the advertisement management system. Theinteraction event type selection controls 931 further include an optionfor selecting e-mail newsletter interactions as a source of userinteraction data. E-mail user interactions can include providing ofe-mails to users (i.e., impressions) and selection of hyperlinkedcontent provided in an e-mail.

The interaction event type selection controls 931 further include anoption for selecting direct to website user interaction data. Direct towebsite user interactions can include a user entering a URL for anadvertiser website into an address bar of a web browser in order to bedirected to the website, or a user selecting a bookmark using a webbrowser in order to be directed to the website.

The interaction event type selection controls 931 further include anoption for selecting user interactions with paid search results as asource of user interaction data. Paid search result user interactionscan include, for example, presentation of paid search results to a useror selection of paid search results by a user. In some implementations,the user of the user interface 920 is allowed to select from amongseveral search engines for which paid search results user interactiondata are to be used for computing conversion path performance measures.

The interaction event type selection controls 931 further include anoption for selecting user interactions with banner ads to be included asa source of user interaction data. Banner ads can include, for example,ads that are presented on non-search related websites. User interactiondata for banner ads can be collected, as described above, using trackingcode embedded in the banner ads (e.g., a single pixel trackingidentifier).

The interaction event type selection controls 931 further includesoptions for selecting other paid ad clicks and other paid ad impressionsas sources of user interaction data. These forms of user interactionscan include all user interactions with ads not included in othercategories displayed as part of the user interface 920.

The interaction event type selection controls 931 further include anoption for selecting affiliate reference user interactions as a sourceof user interaction data. For example, a user can reserve a hotel roomusing a hotel room reservation website. The hotel room reservationwebsite can be an affiliate of a car rental website and present aspecial promotional offer for a discount on car rentals with the carrental website to the user after the user has made the hotel roomreservation. The user selects the promotional offer, but does notimmediately sign up for a car rental. However, the user later returns tothe car rental website and reserves a car (i.e., a conversion occurs).The selection of the promotional offer on the affiliated hotel roomreservation website can be identified as an assist click for theconversion (i.e., the car rental).

The interaction event type selection controls 931 further include anoption for selecting video and audio presentation as a source of userinteraction data. Video presentation can include, for example,presentation of a video relating to a product sold by an advertiser thatis presented by a streaming video website. Audio presentation caninclude, for example, a presentation to a user of an audio clip aboutone or more products offered by an advertiser. The audio clip can bepresented, for example, as part of an Internet radio broadcast. Theinteraction event type selection controls 931 further include optionsfor selecting gadget downloads and interactions as a source of userinteraction data. A gadget can be a user interface that is presented,for example, as part of a webpage, as a stand-alone application, or in apop-up window. The gadget can provide information and receive userinformation.

In some implementations, the interaction event type selection controls931 can include additional controls for selecting other types of contentdistributors (e.g., data source 1 as shown in FIG. 9) that distributecontent associated with one or more advertisers as sources of userinteraction data. The interaction event/source options that have beendiscussed are provided for purposes of example, and other options can beprovided.

As mentioned above, the report area 922 includes conversion pathperformance measures as specified by the user using the selectioncontrols and links of the report area 922. In the example shown, thereport area 922 includes information relating to top paths associatedwith all conversion types for a date range of Mar. 16, 2010 to Apr. 15,2010. The information presented on the report area 922 includes a listof conversion paths 932. The conversion paths 932 include userinteractions for all user interaction types indicated by the interactionevent type selection controls 931. The report area 922 further includesa list of conversion totals 934 indicating a total quantity ofconversions associated with each conversion path for the indicated daterange. For example, the list of conversion totals 934 indicates that aconversion path 936 is associated with 47 conversions for the indicateddate range.

In the example shown, the conversion path 936 includes user interactionswith a travel deals newsletter, a beach vacation banner ad, and a searchresult. The report area 922 indicates that the user interaction with thetravel deals newsletter is a click (i.e., an assist click) and that theuser interaction with the beach vacation banner ad is an impression(i.e., an assist impression). The report area 922 further indicates thatthe search result is a search result for a search term of “vacation” andthat the user interaction with the search result is a click (i.e., alast click).

A conversion path 938 displayed as part of the list of conversion paths932 includes user interactions with first and second banner ads, atravel gadget, and a paid search result. The report area 922 indicatesthat the user interaction with the first banner ad is an impression(i.e., an assist impression) and that the user interaction with thesecond banner ad is a selection (i.e., an assist click). The report area922 further indicates that the user interaction with the travel gadgetis a selection (i.e., an assist click). The travel gadget can be a userinterface that is presented, for example, as part of a webpage, as astand-alone application, or in a pop-up window. The travel gadget canprovide information and receive user information. For example, thetravel gadget can provide real time or near real time information oncurrent travel specials. As another example, the travel gadget can allowa user to input information (such as a destination city, originatingcity, or preferred time range for a trip) in order to provide customizedoffers or other information for the user. In some implementations,entering of data into the travel gadget can be identified as a selectionof the travel gadget (e.g., an assist click or last click).

Still referring to the conversion path 938, the paid search result isindicated as being associated with a targeting keyword of “Miamivacation.” For example, a match for the keyword “Miami vacation” causedthe paid search result to be presented. The paid search result isadditionally indicated as being associated with a last click prior to aconversion.

In some implementations, other conversion path performance measures canbe presented by the user interface 920. Other conversion pathperformance measures can include time lag measures, path lengthmeasures, assist user interaction measures, assisted conversionmeasures, last click measures, first click measures, and otherpreviously described measures or analyses among others.

The report area 922 includes a display type selection control 948 thatenables the user to select one or more formats for presenting conversionpath performance measures on the report area 922. In the example shown,a selection control 949 is selected. The selection control 949 is usedto select a table format for presentation of some or all of theconversion path performance measures presented on the report area 922.In the example shown, the display type selection control 948additionally includes options for pie chart, bar graph, and relativecomparison chart formats.

The report area 922 includes an export control 952 that enables the userto export some or all of the assisted conversion measures displayed onthe report area 922 as well as other data that are not presented (e.g.,user interaction data and/or other computed performance measures). Insome implementations, the export control 952 allows the user to specifyan export format. For example, the user can export the measures as aspreadsheet file. As another example, the user can export the measuresin a word processing document format, a chart, a portable documentformat, an image, or other document formats.

FIG. 9B is a flow chart of an example process 970 for computingconversion path performance measures using user interaction data frommultiple different sources. The process 970 is a process by which userinteraction data are obtained from at least two different userinteraction data sources, where the user interaction data specify userinteractions that are associated with conversions. Using the userinteraction data from at least two of the different sources, conversionpath performance measures are computed and data that cause presentationof the conversion path performance measures are provided to a userdevice.

User interaction data are obtained from multiple data sources (972). Theuser interaction data that are obtained from each of the data sourcesrepresent user interactions that are associated with a conversion. Asdescribed above, user interactions that are performed by a convertinguser for a conversion and during a conversion cycle for the conversioncan be associated with a conversion, for example, by being stored with areference to the conversion (e.g., a conversion identifier) and/or beingindexed according to the conversion (e.g., indexed to the conversionidentifier).

The user interaction data from each of the data sources can specifydifferent types of use interactions. For example, one of the datasources can be a search system that tracks user interactions with searchresults that are provided in response to a search query. Another datasource can be an advertisement management system that tracks userinteractions with advertisements that are provided with search results,as described above. Other data sources can include an analyticsapparatus that tracks user interactions with web pages (and componentsthereof) that are provided, for example, by (or for) an advertiserand/or an advertisement management system that tracks user interactionswith image advertisements (or other advertisements) for an advertiser,where the image advertisements are presented, for example, with webpages provided by a third party content provider.

The user interaction data from at least one of the user interaction datasources can specify, for a conversion, a set of user interactions withcontent items that are associated with one or more targeting keywords.As described in detail above, the targeting keywords are keywords thatcontrol distribution of the content items, for example, with searchresult pages that are provided in response to search queries that arereceived from user devices. For example, an advertisement managementsystem can track user interactions with advertisements that arepresented with search results pages and store user interaction datarepresenting the user interactions in a data store.

In some implementations, the user interaction data from at least oneother user interaction data source specifies, for a conversion, a set ofuser interactions with a web page that is provided by a content providerfor which distribution of content items is controlled using thetargeting keywords described above. For example, as described above, ananalytics apparatus can track user interactions (e.g., page views,widget interactions, and other interactions) that occur with a web pagethat is provided by an advertiser. For example, following presentationof an advertisement on a search results page a user may navigatedirectly to the advertiser's web page of first visit another web pageprior to entering a URL for the advertiser's web page into the user'sbrowser. An analytics apparatus can identify, as a referring resource(i.e., a referring web page) the web page from which the user navigated,or that the user directly entered the URL, and also track subsequentuser interactions with the advertiser's web page. When a user converts(i.e., performs an interaction that constitutes a conversion, theanalytics apparatus can associate a conversion identifier with the userinteraction data for the converting user.

Other user interactions can be tracked by other apparatus, such thatuser interaction data representing these user interactions is available.For example, a content item that is provided directly on or with a webpage (or another network resource) of a third party can include a scriptthat upon presentation and/or selection of the content item cause datato be sent to an apparatus that tracks user interactions with thatcontent item.

A request for conversion path performance measures is received (974). Insome implementations, the request for conversion path performancemeasures can include data specifying two or more types of userinteraction event types for which conversion path performance measuresare to be computed. For example, as described above, a user can select,from a user interface, multiple types of user interactions for which theconversion path performance measures are to be computed. In someimplementations, at least two of the different types of userinteractions are represented by user interaction data from two or moredifferent sources of user interaction data. For example, if the requestfor conversion path performance measures specifies that the measures areto be computed using user interactions with search results and userinteractions with advertisements provided with search results pages,user interaction data from a search system and user interaction datafrom an advertisement management system may be used to compute theconversion path performance measures.

The user interface from which the types of user interactions areselected can be presented by a user device, for example, using data thatcauses presentation of the user interface. In some implementations, thedata cause presentation of event type selection controls in the userinterface, where the event type selection controls enable a user toselect one or more types of user interaction event types for whichconversion path performance measures are to be computed.

An aggregate set of user interaction data are selected from two or moreof the user interaction data sources (976). In some implementations, thetwo or more sources are sources that provide user interaction data forthe selected user interaction event types that were selected from theuser interface.

The conversion path performance measures are computed using theaggregate set of user interaction data (978). The conversion pathperformance measures that are computed can include, for example, atleast one a top paths measure, a path length measure, an assistedconversion measure, an assist interaction measure, and other measuresthat are described above. The conversion path performance measures canbe computed, for example, using the user interaction data from each ofthe different sources. For example, a top paths measure can include eachuser interaction represented by user interaction data and specify atemporal order in which each of the user interactions occurred,irrespective of the sources from which the user interaction data wasobtained.

Data that cause presentation of the conversion path performance measuresare provided (980). In some implementations, the data cause presentationof a user interface in which the conversion path performance measuresare presented. In other implementations, the data can be in a wordprocessing format, a spreadsheet format, or another format that can beprocessed by an application on the user device to cause presentation ofthe conversion path performance measures. Presentation of conversionpath performance measures is described in more detail above.

Processing Pipeline

Overview of the Backend System and Underlying Data Storage Model

As described above, the performance analysis apparatus 120 can include afront end component and a backend data processing pipeline (“backendpipeline”) that facilitate computation and presentation of conversionpath performance measures. The backend pipeline is responsible forgathering raw user interaction data, also referred to as event data,from various information sources (e.g., event logs for impressions,clicks, and conversions, and corresponding spam logs) and processing andconverting raw user interaction data (i.e., initial user interactiondata) into performance measures that are relevant and meaningful to anadvertiser user.

Due to the constant growth and the immensity of the user interactiondata for prior user interactions (also referred to as user interactionevents), the backend pipeline can prepare (i.e., process and/or format)the data in a format and store the data in a suitable data storage modelthat avoids unnecessary duplication of the data, allows easy update withnew data, and facilitates fast and efficient generation of the desiredmeasures based on the data.

FIG. 10A is a diagram of an example data processing pipeline 1000 forproviding the conversion performance reports. In some implementations,the backend pipeline of the performance analysis apparatus can beimplemented using the data processing pipeline 1000. The example dataprocessing pipeline 1000 can be used, for example, to obtain data fromvarious sources, process the data, and analyze the data to generatesearch conversion performance reports, as described above.

The data processing pipeline 1000 extracts or obtains the raw userinteraction data for individual user interaction events from data logs(e.g., impression log 1002, click log 1004, conversion log 1006, etc.)and first organizes the raw user interaction data according to anadvertiser-user centered data storage model (e.g., in a history table1010). Then, the data processing pipeline 1000 extracts the userinteraction data that are stored according to the advertiser-usercentered data model (e.g., in the history table 1010), and reorganizesthe user interaction data according to a conversion centered datastorage model (e.g., in a baseview table 1018). By reorganizing the userinteraction data into a conversion centered data model (e.g., in thebaseview table 1018), all information related to a conversion, such asthe user interaction data of all related user interactions (e.g., priorclicks and impressions), can be easily and efficiently extracted andprocessed for generating various measures in the conversion performancereports.

As an optimization, the raw user interaction data related to eachconversion (also referred to as a conversion event) can be enriched withadditional information from the advertising database (e.g., anadvertising database 1016). For example, values of additionaladvertising parameters can be added to the baseview table 1018. Theseadvertising parameters can be subsequently used as additional filters toseparate out useful subsets of the user interaction data for generatingthe performance reports of interest to advertisers.

As an additional layer of optimization, data tables (e.g., an aggregatetable 1022) including aggregated statistics for specific advertisingparameters can be created and update periodically. These aggregatedtables can be generated based on an estimate of the kinds of measuresand performance reports for which the reports are likely to be requestedmost frequently and are most time-consuming to generate on the fly. Theconsiderations for usage and latency can be balanced againstconsiderations for storage and additional data processing resources todetermine what kinds of pre-aggregation should be performed and storedin the aggregate table.

When the front end reporting system (e.g., a reporting front end 1026 inthe performance analysis apparatus 120 of FIG. 1) receives a request fora conversion performance report, the front end system communicates withan extractor interface (e.g., an extractor 1024) which determineswhether pre-aggregated data suitable for generating and/or providing thereport are available in aggregate tables. If suitable pre-aggregateddata are available, the report is generated using the pre-aggregateddata. If suitable aggregated data are not already available in theaggregate tables (e.g., the aggregate table 1022), the report can begenerated from the user interaction data stored in the conversioncentered data model (e.g., the baseview table 1018).

In a system where an immense amount of data are stored and only a muchsmaller subsection of the data are needed at a time to satisfy aparticular information need, storing the data in a manner thatfacilitates efficient retrieval and processing of selected subsets ofthe stored data can improve the efficiency of the system. In addition,as the large amount of data are likely distributed among a large numberof data servers, a data storage model that enable efficient parallelprocessing of the distributed data on different data servers canincrease quality and speediness of the information service. Adistributed data storage model can be implemented to store the userinteraction data at different stages of the data processing pipeline1000. The distributed data storage model allows parallel processing ofthe data stored on different data servers and efficient joining of theresults provided by a distributed collection of data processingapparatus.

An example distributed data storage model can be implemented as asparse, loosely distributed, persistent multi-dimensional sorted map.The map can be indexed by a row key, a column key, and a timestamp. Thevalues for the row key and the column key can be arbitraryun-interpreted strings. However, a user of the distributed data storagemodel can create schemas for the row key and/or column keys such thatdesired data locality can emerge in the distributed data storage modelwhen key values are generated according to the schemas. In someimplementations, the distributed data storage model need not be sortedin all dimensions for some applications.

In the example distributed data storage model, as a feature to avoidconcurrent updates to the same row, every read or write of data under asingle row key is atomic (i.e., the smallest selectable row unit)irrespective of the number of different columns being read or written inthe row. This feature also allows dynamic partitioning of the data setat different row key values, and allows parallel processing of thedifferent partitions.

The distributed data storage model also allows storage of multipleversions of the same data in each cell of a data table and thesedifferent versions are indexed by timestamp. Although the timestampvalues can be actual time values associated with the data, in certainapplications, other properties associated with the data can also be usedas timestamps to distinguish the different data entities stored in eachcell.

In some implementations, the above example distributed data storagemodel can be used to implement one or more of the history table 1010,the baseview table 1018, and the aggregate table 1022. Other similardistributed data storage model may also be used.

Data Processing through the Backend System

As illustrated in FIG. 10A, the data processing pipeline 1000 begins atone or more data sources, such as the impression log 1002, the click log1004, and the conversion log 1006. The impression log 1002 includes rawuser interaction data for ad impressions that have been served onpublisher properties to consumers. The click log 1004 includes raw userinteraction data for user selections that have been performed withrespect to advertisements served on publisher properties (e.g., searchresult pages, web pages, or other content items containing ad slots).The conversion log 1006 includes raw user interaction data for useractions that constitute conversions (e.g., a purchase, a sign-up forservice, etc.) as defined by an advertiser following user selection ofan advertisements of the advertiser that has been served on a publisherproperty. Other data logs that are utilized in the data processingpipeline can also include spam versions of the impression, click, andconversion logs (e.g., spam impression log, spam click log, spamconversion log), and logs of other user interactions that may be ofinterest to the advertiser.

In some instances, the raw user interaction data for user interactionscan be added to their corresponding data logs in real-time as the userinteractions occur at various consumer, ad server, advertiser, orpublisher sites. In other instances, the raw user interaction data formultiple user interactions can be received from intermediate servers andadded to the data logs at scheduled intervals. For example, the raw userinteraction data for an ad impressions can be recorded in real-time inthe impression log 1002 as a download request for an advertisement isreceived from a client device at which the advertisement will bepresented. For another example, reports of ad impressions served canalso be received periodically from intermediate ad servers and added tothe impression log 1002. As a result, the data logs can continuouslygrow in size as new raw user interaction data are received and recorded.

In some implementations, as the raw user interaction data for new userinteractions are continuously recorded in the data logs, a log importercomponent 1008 periodically, or at scheduled intervals, scans the datalogs to obtain the newly added raw user interaction data. In someimplementations, a separate log importer instance can be implemented foreach log type (e.g., impression, click, conversion, spam impression,spam click, spam conversion, etc.) to perform the extraction of raw userinteraction data from the data log and writing the new data into thehistory table 1010.

The history table 1010 stores the raw user interaction data grouped byadvertiser and user pairs. The history table 1010 can also includecolumns or column families each corresponding to a respective event type(e.g., click, impression, conversion, spam click, spam impression, spamconversion, etc.). Each cell of the history table 1010 can store the rawuser interaction data of multiple user interactions that are associatedwith the corresponding row key value (e.g., advertiser-user pair) andcolumn key value (e.g., event type) of the cell.

In some implementations, each advertiser can be associated with a uniqueadvertiser identifier, such as an advertising account ID, or otherstrings or signatures that uniquely identify the advertiser oradvertising account. Each user can be associated with a unique useridentifier, such as a device identifier, an anonymized IP address, auser account ID, or other strings that uniquely identify each userclient device or consumer. In some implementations, user interactiondata of user actions that are performed on the client device would beassociated with the cookie ID for a cookie issued to the client device,and the cookie ID can be used as the unique user identifier for the userinteraction data.

In some implementations, for the raw user interaction data of each userinteraction, the log importer 1006 extracts the advertiser identifierand the user identifier associated with the user interaction from, forexample, one or more fields of the raw user interaction data. The logimporter 1006 then generates a unique string pattern corresponding tothe advertiser identifier and user identifier pair. For example, aschema for the row key of the history table 1010 can include a leadingstring segment corresponding to an advertiser_id followed by a user_id.

In some implementations, if the advertiser identifier and useridentifier pair already has a corresponding row in the distributedhistory table 1010, the raw user interaction data of the userinteraction is written to a cell in the row that corresponds to theevent type associated with the user interaction (e.g., a click, animpression, a conversion, a spam click, a spam impression, or a spamconversion). If the advertiser identifier and user identifier pair doesnot already have a corresponding row in the distributed history table1010, a new row corresponding to this advertiser identifier useridentifier pair is created and inserted into the distributed historytable 1010. The raw user interaction data of the user interaction iswritten to a cell in the row that corresponds to the event typeassociated with the user interaction. Since the distributed historytable 1010 are sorted according to its row key values (i.e., advertiseridentifier and user identifier pair), raw user interaction dataassociated with the same advertiser (but different users) can be storedin consecutive rows.

In each cell of the history table 1010, the raw user interaction datafor each user interaction is stored with a corresponding timestampindicating the time at which the user interaction occurred. Therefore,the cell corresponding to an impression event type would include allrecorded impressions that have occurred for the advertiser-user pair.Similarly, the cell corresponding to a click event type would includeall recorded clicks that have occurred for the advertiser-user pair.Similarly, the cell corresponding to a conversion type (or a subtype ofthe conversion type) would include all recorded conversions (or thesubtype of conversions) that have occurred for the advertiser-user pair.

In some implementations, the raw user interaction data already includesthe timing information in one of its fields, and the value in the rawuser interaction data can be extracted and converted to a correspondingtimestamp value. In some implementations, the timing information for auser interaction needs to be obtained from other sources (e.g., from anad server or other third party sources) based on an event identifierincluded the raw user interaction data. The timestamps associated withthe user interaction data stored in each cell allow filtering of datawithin a specified time range (e.g., 30 days from last update, betweendate x and date y, etc.).

In some implementations, in addition to advertiser identifier, useridentifier, and timestamp, the raw user interaction data for each userinteraction can include information fields identifying the content item(e.g., the advertisement or creative) presented to and/or selected bythe user, the web address for the publisher's property, a last clickleading to a conversion, a delivery period for the advertisement, an adgroup for the advertisement served, a campaign for the advertisementserved, and so on. In some implementations, these information fields arenormalized as identifiers to save resources used to store and transmitthe information. For example, the raw user interaction data in the datalogs and history table 1010 can include only the information in theirnormalized form.

The history table 1010 includes the raw user interaction data ofpreviously occurred user interactions extracted from the data logs, andgrouped by advertiser and user identifier pairs. Periodically or atspecified times, an event joining component 1012 queries the historytable 1010 to determine whether raw user interaction data of any newconversions have been added since the last query. If a new conversionhas been added, the event joining component 1012 creates a new row inthe baseview table 1018 for the newly added conversion and stores userinteraction data for the new conversion in the newly added row. Whenbuilding the baseview table 1018 initially, all conversions in thehistory table can be considered new conversion. In some implementations,only the most recent conversions (e.g., last 30, 60 or 90 days) areidentified (e.g., by their timestamps used in this history table 1010)and used to create rows in the baseview table 1018.

The baseview table 1018 stores the raw user interaction data grouped(i.e., indexed) by conversions. Each row of the baseview table 1018includes the raw user interaction data of all prior user interactionsthat occurred within a specified time period prior to the conversion andare associated with the same advertiser user pair as the conversion.Because all user interactions associated with the same advertiser-userpair are stored in the same row in the history table 1010, extractionand processing of the data for the advertiser-user pair can be performedindependently of the extraction and processing of the data for otheradvertiser-user pairs. Therefore, parallel processing and scalability ofthe data storage and processing can be accomplished.

In some implementations, a unique conversion identifier is generated forthe conversion and used as the row key value for the conversion in thebaseview table 1018. In some implementations, the conversion identifierincludes concatenated string segments representing an advertiseridentifier, a conversion type identifier, a date of the conversion, andso on. By generating the row key of the baseview table 1018 according tothis particular schema, conversions pertaining to the same advertiserscan be organized into consecutive rows in the baseview table 1018.Similarly, within the consecutive rows of a particular advertiser,conversions of the same type can be organized into consecutive rows aswell. Furthermore, conversions of the same type under the sameadvertiser are also sorted according to their respective dates ofoccurrence.

By organizing the rows of the baseview table 1018 in this matter,subsections of the baseview table 1018 pertaining to individualadvertisers, conversion types, and/or date ranges can be distributedamong a relatively small number of data servers, and can be extractedand processed in parallel by a distributed system of data processingapparatus. Other schema of the conversion identifier can be implementedto suit various data extraction and processing needs.

The baseview table 1018 also includes columns or column families eachcorresponding to a respective parameter that is associated with theconversion and other user interactions that have occurred within apredetermined time period prior to the conversion. Examples of theparameters can include interaction types such as clicks, impressions,and conversions. In some implementations, conversion type can besubdivided into additional subtypes (e.g., a subtype for purchase, asubtype for sign-up, etc.). For each event type, user interaction datafor all user interaction events of that event type and having occurredwithin the specified time period prior to the conversion are stored withtheir respective timestamps in the same cell corresponding to theparticular interaction type. The user interaction data stored in thesame cell can further be sorted by their timestamps.

For example, for a new conversion that is discovered in the historytable 1010, a new row is created in the baseview table. The userinteraction data representing the clicks stored in the same row as theconversion event and having occurred within a specified time window(e.g., 30 days) prior to the conversion event can be written into a cellcorresponding to clicks in the newly created row of the baseview table1018. The user interaction data representing the clicks are sortedaccording to their associated timestamps in the cell, and form a timesequence of clicks leading to the conversion event. The datarepresenting each click can include and/or be associated with areference to the keyword that caused presentation of the clicked contentitem.

Similarly, data representing impressions that occurred within aspecified time window prior to the new conversion event can be writteninto a cell corresponding to impressions in the new row and form a timesequence of impressions leading to the conversion event. The datarepresenting each impression can include and/or be associated with areference to the keyword that caused presentation of the content item.Other conversions that occur within a time window of the new conversioncan be also written into a cell corresponding to conversions in the newrow and form a time sequence of conversions leading to the newconversion event.

By processing each row of the advertiser-user centered data in thehistory table 1010 and reorganizing the user interaction data in eachrow into one or more conversion centered rows in the baseview table 1018(generally, referred to as “pivoting of the data”), additionalprocessing of each row of conversion centered user interaction data canbe performed independently and in parallel of other rows of theconversion centered user interaction data. Because the backend servesconversion related measures, the transformations from individual userinteraction data to advertiser-user centered data, and then toconversion center data allows fine-grained parallel processing based onindividual conversions.

In some implementations, columns or column families corresponding toother parameters associated with the new conversion and other associateduser interaction data stored in the same row as the new conversion canbe created in the baseview table 1018. Examples of such parametersinclude ad campaign, ad group, keyword, creative, publisher, publisherproperty, advertising criteria, and so on. In some implementations,because the raw user interaction data for user interaction events likelyonly include identifiers for values of these parameters (e.g., acampaign identifier, an ad group identifier, a creative identifier,etc.), the event joining component 1012 can further interface with adatabase joining component 1014 to map the identifiers to theircorresponding data names (e.g., campaign name, ad group name, creativename, etc.).

In some implementations, the database joining component 1014 looks upthe data names by their identifiers in the advertising database 1016which stores the advertiser account and campaign specificationinformation. The database joining component 1014 can supply results ofthe lookups to the event joining component 1012, which in turn fills thecorresponding cells of these parameters with the returned values. Eachof these values can also be associated with their correspondingtimestamps for the corresponding user interaction event to which thevalues pertain.

By enriching the data in the baseview table 1018 with the additionalparameters, a larger variety of performance measures can be generatedbased on the parameter values stored in the baseview table 1018. Forexample, additional filters (e.g., campaign filter, keyword filter, adgroup filter, etc.) can be created based on values for one or more ofthese parameters, and desired subsets of data can be isolated out bytheir filters for additional processing.

The conversion centered data in the baseview table 1018 can be used togenerate conversion performance reports such as those described above.For example, suppose an advertiser is requesting an assist click reportfor all conversions that occurred during a particular time window.First, all rows associated with the particular advertiser's identifiercan be identified from the baseview table 1018 by their respective rowkey values, provided that the row key schema includes a leading segmentcorresponding to the advertiser identifier. These rows can bedistributed to a number of data processing apparatus for processing. Foreach row, clicks stored within the row and having timestamps within thespecified time window are counted by a corresponding data processingapparatus that has been assigned to process that row. One count for thelast click before the conversion can be removed from the total count ofclicks to arrive at the assist click count for the conversion. The dataprocessing apparatus processing each row can return the results and theresults are aggregated to show the quantity of conversions havingvarious assist click counts associated therewith. For example, theassist clicks report can show that 80,000 conversions have one assistclick, 50,000 conversions have two assist clicks, 30,000 conversionshave three assist clicks, and so on. Other reports are possible.

Other examples of performance measures include assist interactionmeasures, path length measures, time lag measures, assisted conversionmeasures, last click measures, first click measures, top path measures,and other conversion path performance measures. Conversion performancereports for individual advertiser accounts can be delivered to accordingto schedules or upon advertiser requests. Reports can be generated for aspecific time period, a specific keywords, creatives, publisher sites,product line, ad groups, ad campaigns, accounts, and so on.

In some implementations, when the reporting front end component 1026receives an advertiser's request for a report, the reporting front end1026 can pass the advertiser's identifier and the requested parametersfor the report to an extractor component 1024. In some implementations,the extractor component 1022 can interface with the baseview table 1018directly to perform the extraction and processing of the desired report.

In some implementations, aggregated tables can be created prior to arequest, such that the extractor can query the existing aggregate tableto obtain the desired reports quickly. The aggregate table 1022 can begenerated by an aggregating component 1020. The aggregating component1020 can run periodically to update the aggregate table 1022 with newdata that have been added to the baseview table 1018 and to discard olddata to save storage space. The types of tables prepared and stored inthe aggregate table 1020 can be determined based on estimates of whatreports may be requested most frequently and the latency for generatingthe reports on the fly.

In some implementations, the aggregate table 1020 can also be stored ina distributed data model such as the example distributed data modeldescribed above. The aggregate table 1020 can be indexed by concatenatedstring segments representing an advertiser identifier and a table type.Other appropriate row key naming schemas for the aggregate table 1020are possible.

Example Processes for the Backend Data Processing

The distributed data storage models described herein are suitable forparallel processing of the data stored therein. For example, the historytable 1010, the baseview table 1018, and the aggregate table 1022, canbe serve as both input and output to a distributed data processingsystem. An example of a distributed data processing system can beimplemented according to a MapReduce framework. However, other suitabledistributed data processing systems may be used as well.

FIG. 10B is a flow chart of an example process 1100 for organizingindividual user interaction data first into an advertiser-user centereddata storage model and then into a conversion centered data storagemodel. The process 1100 can be performed by a distributed network ofdata processing apparatus.

The process 1100 with one or more user interaction logs being processedto obtain respective user interaction data for one or more newlyperformed user interaction events (1102). Then, a first distributed datatable can be updated using the respective user interaction data for thenewly occurred user interaction events (1104). In some implementations,the first distributed data table includes rows indexed and sorted bystrings representing unique pairs of advertiser identifier and useridentifier that are associated with stored user interaction events inthe first distributed data table. Furthermore, each row includes aplurality of ordered cells, and each cell corresponds to a respectiveevent type and containing the user interaction data for stored userinteraction events of said event type. In some implementations, the userinteraction data for each user interaction event is stored with arespective timestamp associated with the user interaction event. Anexample of the first distributed data table is the history table 1010 asshown in FIG. 10A. The processing and updating can be performed by thelog importer component 1006. The processing and updating can beperformed periodically.

After the first distributed data table has been created and updated, acomponent (e.g., by the event joining component 1012 shown in FIG. 10A)can query the first distributed data table to identify conversion eventsthat have occurred within a specified time window (1106). The query canspecify a filter for conversion events and for the time window, and eachrow of the first distributed data table (e.g., the history table 1010 ofFIG. 10A) can be scanned based on event types and timestamps of thestored user interaction events. One or more conversion events can beidentified as a result of the query.

In response to the query, for each identified conversion event, thecomponent receives the user interaction data of the conversion event andrespective user interaction data of all stored user interaction eventsassociated with a same advertiser identifier and user identifier pair(i.e., associated with the same advertiser-user pair) as the conversionevent and having occurred within a specified time period prior to theidentified conversion event (1108). Because user interaction data thatare related to the same advertiser-user pair are stored in the same rowin the first distributed data table (e.g., the history table 1010), onlythat one row needs to be processed to obtain all click events andimpression events having occurred within a specified time period priorto an identified conversion event.

After the user interaction data are received for one or more identifiedconversion event, a new row can be created for each identifiedconversion event in a second distributed data table (1110). The new rowin the second distributed data table can be indexed with a uniqueconversion identifier. The new row can include all of the received userinteraction data for the identified conversion. An example of the seconddistributed data table is the baseview table 1018 shown in FIG. 10A. Thequerying, receiving, and creating can be performed by a distributedsystem of data processing apparatus, such as the event joining component1014 shown in FIG. 10A.

FIG. 10C is a flow chart of an example process 1200 for adding new datain a conversion centered data storage model. For example, when creatinga new row for each identified conversion in the second distributed datatable, additional data related to the user interaction data received forthe identified conversion event can be obtained (1202). In someimplementations, the additional data can be obtained by the databasejoining component 1014 from the advertising database 1016 shown in FIG.10A. The additional data can include, for example, campaign names, adgroup names, keywords, creative names, advertising criteria, and so on.After the additional information is obtained by the database joiningcomponent, for example, one or more cells of the new row can bepopulated using the obtained additional data (1204). This is a “datawidening” process which allows additional filter parameters to becreated to allow generation of desired measures (e.g., ad campaignspecific, ad group specific, creative specific, keyword specific, and/orcriteria specific analysis of the data).

In some implementations, rows of the second distributed data table(e.g., the baseview table 1018) are indexed and sorted by conversionidentifiers. When creating the new row for each identified conversionevent in the second distributed data table, the unique conversionidentifier can be generated for the identified conversion event based ona predetermined schema such that rows associated with conversion eventsof a same advertiser are stored in consecutive rows in the seconddistributed data table (1206). In one example, the unique identifier caninclude a leading segment corresponding to the advertiser identifier,followed by segments corresponding to conversion type and a date.

In some implementations, each row of the second distributed data tableincludes a plurality of ordered cells each corresponding to a respectiveparameter that is relevant to an advertiser's query regarding the userinteraction events stored in the row. Examples of the parametersinclude, click event type, impression event type, conversion event type,ad campaign, ad group, creative, keyword, and so on. The cells areordered because the columns of the second distributed table are fixed inorder.

In some implementations, each cell of the second distributed data tableis operable for storing one or more values for the cell's correspondingparameter, each value being stored with a respective timestampassociated with the value. For example, user interaction datarepresenting the clicks that are associated with a conversion event,such as all the clicks that occurred within a specified time windowprior to the conversion event, can be stored in a single cellcorresponding to the impression event type in the row corresponding tothe conversion event. The user interaction data of each of the clickscan be stored with an associated timestamp indicating the time that theimpression has occurred. The user interaction data of the clicks can besorted according to their associated timestamps, such that a timesequence of clicks can be formed in the cell. In some implementations,clicks that appear to be duplicates (e.g., consecutive clicks on thesame ad during the same user session) can be collapsed into a singleclick event in the cell.

FIG. 10D is a flow chart of an example process 1300 for generatingaggregated data utilizing the conversion centered data storage model.

In some implementations, an aggregating component (e.g., the aggregatingcomponent 1020) can query the second distributed table (e.g., thebaseview table 1018) to obtain a subset of the stored user interactiondata in one or more rows of the second distributed table (1302). Then,for rows associated with a same advertiser identifier, the stored valuesfor a specified parameter can be aggregated (1304). A new row can becreated in a third distributed data table (e.g., the aggregate table1022), wherein rows in the third distributed data table are indexed bythe advertiser identifier and the specified parameter. For example, theaggregate table can include a row for an advertiser and a keyword pair.The keyword can be one of the keywords that the advertiser uses totargeting search ads. The row can have a row key that includes a leadingsegment corresponding to the advertiser identifier, followed by thekeyword name (e.g., “shoe”). Within that row, aggregated userinteraction data that are associated with advertiser-keyword pair can bestored. For example, a cell corresponding to the click event type canstore all the clicks that have occurred for an ad presented due to thekeyword. Other types of aggregate tables can be generated depending onthe information needs of the advertisers.

FIG. 10E is an example flow chart of an example process 1400 forproviding conversion performance reports based on pre-stored aggregatedata. In some implementations, a request for aggregated user interactiondata associated with on one or more parameters are received (1402). Therequest can be associated with a specified advertiser identifier. Acomponent queries the third distributed data table (e.g., the aggregatetable 1022) based on the one or more parameters and the specifiedadvertiser identifier (1404). In cases where suitable pre-stored dataare available in the third distributed data table, the requestedaggregated user interaction data can be returned based on the one ormore parameters and the specified advertiser identifier

Example Computer

FIG. 11 is a block diagram of an example computer system 500 that can beused to compute conversion path performance measures and provide aconversion performance report. The system 1100 includes a processor1110, a memory 1120, a storage device 1130, and an input/output device1140. Each of the components 1110, 1120, 1130, and 1140 can beinterconnected, for example, using a system bus 1150. The processor 1110is capable of processing instructions for execution within the system1100. In one implementation, the processor 1110 is a single-threadedprocessor. In another implementation, the processor 1110 is amulti-threaded processor. The processor 1110 is capable of processinginstructions stored in the memory 1120 or on the storage device 1130.

The memory 1120 stores information within the system 1100. In oneimplementation, the memory 1120 is a computer-readable medium. In oneimplementation, the memory 1120 is a volatile memory unit. In anotherimplementation, the memory 1120 is a non-volatile memory unit.

The storage device 1130 is capable of providing mass storage for thesystem 1100. In one implementation, the storage device 1130 is acomputer-readable medium. In various different implementations, thestorage device 1130 can include, for example, a hard disk device, anoptical disk device, or some other large capacity storage device.

The input/output device 1140 provides input/output operations for thesystem 1100. In one implementation, the input/output device 1140 caninclude one or more of a network interface device, e.g., an Ethernetcard, a serial communication device, e.g., and RS-232 port, and/or awireless interface device, e.g., an 802.11 card. In anotherimplementation, the input/output device can include driver devicesconfigured to receive input data and send output data to otherinput/output devices, e.g., keyboard, printer and display devices 1160.Other implementations, however, can also be used, such as mobilecomputing devices, mobile communication devices, set-top box televisionclient devices, etc.

The performance analysis apparatus 120 can be realized by instructionsthat upon execution cause one or more processing devices to carry outthe processes and functions described above. The performance analysisapparatus 120 and/or advertisement management system 110 can beimplemented in a distributed manner over a network, such as a serverfarm, or can be implemented in a single computer device.

Although an example processing system has been described in FIG. 11,implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in other types ofdigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A method performed by a data processingapparatus, the method comprising: detecting, by the data processingapparatus and based at least in part on data generated by execution oftracking code included in various web pages, user interactions with thevarious web pages; storing user interaction data representing thedetected user interactions; identifying, from the stored userinteraction data, conversions that were performed by users; identifying,for each of at least some of the conversions, a last clicked contentitem that was clicked within a specified period prior to a particularconversion; identifying, as assist keywords for each of at least some ofthe conversions, one or more keywords that, prior to the specifiedperiod, caused presentation of a content item to a user that performedthe conversion; determining, by one or more data processing apparatus, atotal number of assisted conversions for each of the assist keywords,the total number of assisted conversions for a particular keyword beinga total number of the conversions performed by multiple different usersthat, according to the user interaction data, were each presented acontent item using the particular assist keyword prior to the specifiedperiod; determining, for each particular assist keyword from the atleast one of the assist keywords and by one or more data processingapparatus, a ratio between the total number of assisted conversions forthe particular assist keyword and a total number of assisted conversionsfor multiple different assist keywords; receiving conversion values thatare associated with the conversions, wherein receiving conversion valuesthat are associated with the conversions comprises receiving monetaryvalues of the conversions; and determining, for each of the assistkeywords and using the received conversion values, an assistedconversion value for the assist keyword; and providing a web pageincluding an assisted conversion report including assisted conversionmeasures quantifying, for at least some of the assist keywords, thedetermined ratio and the assisted conversion value.
 2. The method ofclaim 1, further comprising receiving a request for an assistedconversions report that specifies an assisted conversion measure for acontent distribution campaign.
 3. The method of claim 1, whereinproviding the web page including the assisted conversion reportcomprises providing data that cause presentation of the assistedconversion measure for a content distribution campaign, wherein theassisted conversion measure for the content distribution campaign is anaggregate measure of assisted conversions for two or more of the assistkeywords for the content distribution campaign.
 4. The method of claim3, wherein providing data that cause presentation of the assistedconversions measure further comprises providing data that causepresentation of the assisted conversion measure for each of the two ormore assist keywords in the content distribution campaign.
 5. The methodof claim 1, wherein providing the web page including the assistedconversion report comprises: computing a cumulative quantity of assistedconversions over a specified period, wherein each assisted conversion isa conversion that occurred during the specified period; and providingdata that present the cumulative quantity in the web page.
 6. The methodof claim 1, wherein providing the web page including the assistedconversion report comprises: computing, for each of the assist keywords,a quantity of click-assisted conversions for the assist keyword, a clickassisted conversion for an assist keyword being a conversion for whichthe assist keyword caused presentation of a content item that wasclicked by a converting user; and providing data that presents thequantity of click-assisted conversions in the web page.
 7. The method ofclaim 6, wherein providing the web page including the assistedconversion report comprises: computing, for each of the assist keywords,a ratio of the quantity of click-assisted conversions relative to aquantity of last click conversions for the assist keyword, wherein thequantity of last click conversions for the assist keyword is a quantityof conversions for which the keyword caused presentation of a contentitem that was last clicked prior to the conversion; and providing datathe computed ratio in the web page.
 8. The method of claim 1, whereinproviding the web page including the assist conversion report comprises:computing, for each of the assist keywords, a quantity ofimpression-assisted conversions for the assist keyword, an impressionassisted conversion for an assist keyword being a conversion for whichthe assist keyword caused presentation of a content item that waspresented to a converting user; and providing data that present thequantity of impression-assisted conversions in the web page.
 9. Themethod of claim 8, wherein providing the web page including the assistedconversion report comprises: computing, for each of the assist keywords,a ratio of the quantity of impression-assisted conversions relative to aquantity of last click conversions for the assist keyword, wherein thequantity of last click conversions for the assist keyword is a quantityof assisted conversions for which the assist keyword caused presentationof a content item that was last click prior to the conversion; andproviding data the presented the computed ratio in the web page.
 10. Asystem comprising: a data store; and one or more computers configured tointeract with the data store and further configured to: detecting, basedat least in part on data generated by execution of tracking codeincluded in various web pages, user interactions with the various webpages; storing, in the data store, user interaction data representingthe detected user interactions; identifying, from the stored userinteraction data, conversions that were performed by users; identifying,for each of at least some of the conversions, a last clicked contentitem that was clicked within a specified period prior to a particularconversion; identify, as assist keywords for each of at least some ofthe conversions, one or more keywords that, prior to the specifiedperiod, caused presentation of a content item to a user that performedthe conversion; determine a total number of assisted conversions foreach of the assist keywords, the total number of assisted conversionsfor a particular keyword being a total number of the conversionsperformed by multiple different users that, according to the userinteraction data, were each presented a content item using theparticular assist keyword prior to the specified period; and determine,for each particular assist keyword from the at least one of the assistkeywords, a ratio between the total number of assist conversions for theparticular assist keyword and a total number assisted conversions formultiple different assist keywords; and receiving conversion values thatare associated with the conversions, wherein receiving conversion valuesthat are associated with the conversions comprises receiving monetaryvalues of the conversions; and determining, for each of the assistkeywords and using the received conversion values, an assistedconversion value for the assist keyword; and providing a web pageincluding an assisted conversion report including an assisted conversionmeasure quantifying, for at least some of the assist keywords, thedetermined ratio and the assisted conversion value.
 11. The system ofclaim 10, further comprising a user device, wherein the one or morecomputers are further configured to interact with the user device, andwherein the one or more computers are configured to provide the datathat cause presentation of the assisted conversion report to the userdevice.
 12. The system of claim 10, wherein the one or more computersare further configured to: receive a request for an assisted conversionsreport that specifies an assisted conversion measure for a contentdistribution campaign; and provide, in response to the request, datathat cause presentation of the web page including the assistedconversions report and an aggregate measure of assisted conversions fortwo or more of the assist keywords for the content distributioncampaign.
 13. The system of claim 12, wherein the one or more computersare further configured to provide data that cause presentation of theweb page including the assisted conversion measure for each of the twoor more assist keywords in the content distribution campaign.
 14. Thesystem of claim 10, wherein the one or more computers are furtherconfigured to: compute a cumulative quantity of assisted conversionsover a specified period, wherein each assisted conversion is aconversion that occurred during the specified period; and provide datathat present the cumulative quantity in the web page.
 15. The system ofclaim 10, wherein the one or more computers are further configured toobtain data specifying conversion indexed user interaction data for theconversions, wherein the conversion indexed user interaction data for aparticular conversion is indexed according to a conversion identifierthat represents the particular conversion, and wherein the userinteraction data that are indexed to the particular conversion areassociated with a single user identifier.
 16. The system of claim 10,wherein the one or more computers are further configured to: compute,for each of the assist keywords, a quantity of click-assistedconversions for the assist keyword, a click assisted conversion for anassist keyword being a conversion for which the assist keyword causedpresentation of a content item that was selected by a converting user;and provide data that present the quantity of click-assisted conversionsin the web page.
 17. The system of claim 16, wherein the one or morecomputers are further configured to: compute, for each of the assistkeywords, a ratio of the quantity of click-assisted conversions relativeto a quantity of last click conversions for the assist keyword, whereinthe quantity of last click conversions for the assist keyword is aquantity of conversions for which the keyword caused presentation of acontent item that was last clicked prior to the conversion; and providedata the presented the computed ratio in the web page.
 18. The system ofclaim 10, wherein the one or more computers are further configured to:compute, for each of the assist keywords, a quantity ofimpression-assisted conversions for the assist keyword, an impressionassisted conversion for an assist keyword being a conversion for whichthe assist keyword caused presentation of a content item that waspresented to a converting user; and provide data that present thequantity of impression-assisted conversions in the web page.
 19. Anon-transitory computer storage medium encoded with a computer program,the program comprising instructions that when executed by one or moredata processing apparatus cause the one or more data processingapparatus to perform operations comprising: detecting, based at least inpart on data generated by execution of tracking code included in variousweb pages, user interactions with the various web pages; storing userinteraction data representing the detected user interactions;identifying, from the stored user interaction data, conversions thatwere performed by users; identifying, for each of at least some of theconversions, a last clicked content item that was clicked within aspecified period prior to a particular conversion; identifying, asassist keywords for each of at least some of the conversions, one ormore keywords that, prior to the specified period, caused presentationof a content item to a user that performed the conversion; determining atotal number of assisted conversions for each of the assist keywords,the total number of assisted conversions for a particular keyword beinga total number of the conversions performed by multiple different usersthat, according to the user interaction data, were each presented acontent item using the particular assist keyword prior to the specifiedperiod; determining, for each particular assist keyword from the atleast one of the assist keywords, a ratio between the total number ofassisted conversions for the particular assist keyword and a totalnumber of assisted conversions for multiple different assist keywords;and receiving conversion values that are associated with theconversions, wherein receiving conversion values that are associatedwith the conversions comprises receiving monetary values of theconversions; and determining, for each of the assist keywords and usingthe received conversion values, an assisted conversion value for theassist keyword; and providing a web page including an assistedconversion report including assisted conversion measures quantifying,for at least some of the assist keywords, the determined ratio and theassisted conversion value.